Intuitive computing methods and systems

ABSTRACT

A smart phone senses audio, imagery, and/or other stimulus from a user&#39;s environment, and acts autonomously to fulfill inferred or anticipated user desires. In one aspect, the detailed technology concerns phone-based cognition of a scene viewed by the phone&#39;s camera. The image processing tasks applied to the scene can be selected from among various alternatives by reference to resource costs, resource constraints, other stimulus information (e.g., audio), task substitutability, etc. The phone can apply more or less resources to an image processing task depending on how successfully the task is proceeding, or based on the user&#39;s apparent interest in the task. In some arrangements, data may be referred to the cloud for analysis, or for gleaning. Cognition, and identification of appropriate device response(s), can be aided by collateral information, such as context. A great number of other features and arrangements are also detailed.

RELATED APPLICATION DATA

This application is a division of application Ser. No. 12/797,503, filedJun. 9, 2010 (published as 20110161076), which claims priority toprovisional applications 61/318,217, filed Mar. 26, 2010, 61/315,475,filed Mar. 19, 2010, and 61/291,812, filed Dec. 31, 2009. Thisapplication is also a continuation-in-part of application Ser. No.13/708,434, filed Dec. 7, 2012 (published as 20130128060), which is adivision of application Ser. No. 13/401,332, filed Feb. 21, 2012 (nowU.S. Pat. No. 8,422,994), which is a division of application Ser. No.12/712,176, filed Feb. 24, 2010 (now U.S. Pat. No. 8,121,618), which isa continuation-in-part of application Ser. No. 12/640,386, filed Dec.17, 2009 (now U.S. Pat. No. 8,175,617), which claims priority toprovisional applications 61/255,817, filed Oct. 28, 2009; 61/261,028,filed Nov. 13, 2009; 61/263,318, filed Nov. 20, 2009; 61/264,639, filedNov. 25, 2009; 61/266,965, filed Dec. 4, 2009; and 61/285,726, filedDec. 11, 2009.

This specification concerns extensions and improvements to technologydetailed in the assignee's previous patents and patent applications,including U.S. Pat. No. 6,947,571, and U.S. application Ser. No.12/716,908, filed Mar. 3, 2010 (now U.S. Pat. No. 8,412,577); Ser. No.12/695,903, filed Jan. 28, 2010 (now U.S. Pat. No. 8,433,306); PCTapplication PCT/US09/54358, filed Aug. 19, 2009 (published asWO2010022185, which has been nationalized as U.S. application Ser. No.13/011,618, published as 20110212717); Ser. No. 12/490,980, filed Jun.24, 2009 (published as 20100205628); Ser. No. 12/484,115, filed Jun. 12,2009 (published as 20100048242); and Ser. No. 12/271,772, filed Nov. 14,2008 (published as 20100119208).

The principles and teachings from these just-cited documents areintended to be applied in the context of the presently-detailedarrangements, and vice versa. (The disclosures of the above-citedpatents and applications are incorporated by reference, as if set forthherein in their entireties.)

TECHNICAL FIELD

The present specification concerns a variety of technologies; mostconcern enabling smart phones and other mobile devices to respond to theuser's environment, e.g., by serving as intuitive hearing and seeingdevices.

INTRODUCTION

Cell phones have evolved from single purpose communication tools, tomulti-function computer platforms. “There's an app for that” is afamiliar refrain.

Over two hundred thousand applications are available for smartphones—offering an overwhelming variety of services. However, each ofthese services must be expressly identified and launched by the user.

This is a far cry from the vision of ubiquitous computing, dating backover twenty years, in which computers demand less of our attention,rather than more. A truly “smart” phone would be one that takesactions—autonomously—to fulfill inferred or anticipated user desires.

A leap forward in this direction would be to equip cell phones withtechnology making them intelligent seeing/hearing devices—monitoring theuser's environment and automatically selecting and undertakingoperations responsive to visual and/or other stimulus.

There are many challenges to realizing such a device. These includetechnologies for understanding what input stimulus to the devicerepresents, for inferring user desires based on that understanding, andfor interacting with the user in satisfying those desires. Perhaps thegreatest of these is the first, which is essentially the long-standingproblem of machine cognition.

Consider a cell phone camera. For each captured frame, it outputs amillion or so numbers (pixel values). Do those numbers represent a car,a barcode, the user's child, or one of a million other things?

Hypothetically, the problem has a straightforward solution. Forward thepixels to the “cloud” and have a vast army of anonymous computers applyevery known image recognition algorithm to the data until one finallyidentifies the depicted subject. (One particular approach would be tocompare the unknown image with each of the billions of images posted toweb-based public photo repositories, such as Flickr and Facebook. Afterfinding the most similar posted photo, the descriptive words, or“meta-data,” associated with the matching picture could be noted, andused as descriptors to identify the subject of the unknown image.) Afterconsuming a few days or months of cloud computing power (and megawattsof electrical power), an answer would be produced.

Such solutions, however, are not practical—neither in terms of time orresources.

A somewhat more practical approach is to post the image to acrowd-sourcing service, such as Amazon's Mechanical Turk. The servicerefers the image to one or more human reviewers, who provide descriptiveterms back to the service, which are then forwarded back to the device.When other solutions prove unavailing, this is a possible alternative,although the time delay is excessive in many circumstances.

In one aspect, the present specification concerns technologies that canbe employed to better address the cognition problem. In one embodiment,image processing arrangements are applied to successively gain more andbetter information about the input stimulus. A rough idea of an image'scontent may be available in one second. More information may beavailable after two seconds. With further processing, still more refinedassessments may be available after three or four seconds, etc. Thisprocessing can be interrupted at any point by an indication—express,implied or inferred—that the user does not need such processing tocontinue.

If such processing does not yield prompt, satisfactory results, and thesubject of the imagery continues to be of interest to the user (or ifthe user does not indicate otherwise), the imagery may be referred tothe cloud for more exhaustive, and lengthy, analysis. A bookmark orother pointer may be stored on the smart phone, allowing the user tocheck back and learn the results of such further analysis by the remoteservice. Or the user can be alerted if such further analysis reaches anactionable conclusion.

Cognition, and identification of appropriate device response(s), can beaided by collateral information, such as context. If the smart phoneknows from stored profile information that the user is a 35 year oldmale, and knows from GPS data and associated map information that theuser is located in a Starbucks in Portland, and knows from time andweather information that it is a dark and snowy morning on a workday,and recalls from device history that in several prior visits to thislocation the user employed the phone's electronic wallet to buy coffeeand a newspaper, and used the phone's browser to view websites reportingfootball results, then the smart phone's tasks are simplifiedconsiderably. No longer is there an unbounded universe of possible inputstimuli. Rather, the input sights and sounds are likely to be of typesthat normally would be encountered in a coffee shop on a dark and snowymorning (or, stated conversely, are not likely to be, e.g., the sightsand sounds that would be found in a sunny park in Tokyo). Nor is therean unbounded universe of possible actions that are appropriate inresponse to such sights and sounds. Instead, candidate actions arelikely those that would be relevant to a 35 year old,football-interested, coffee-drinking user on his way to work in Portland(or, stated conversely, are not likely to be the actions relevant, e.g.,to an elderly woman sitting in a park in Tokyo).

Usually, the most important context information is location. Second-mostrelevant is typically history of action (informed by current day ofweek, season, etc). Also important is information about what otherpeople in the user's social group, or in the user's demographic group,have done in similar circumstances. (If the last nine teenage girls whopaused at a particular location in Macys captured an image of a pair ofboots on an aisle-end display, and all were interested in learning theprice, and two of them were also interested in learning what sizes arein stock, then the image captured by the tenth teenage girl pausing atthat location is also probably of the same pair of boots, and that useris likely interested in learning the price, and perhaps the sizes instock.) Based on such collateral information, the smart phone can loadrecognition software appropriate for statistically likely stimuli, andcan prepare to undertake actions that are statistically relevant inresponse.

In one particular embodiment, the smart phone may have availablehundreds of alternative software agents—each of which may be able toperform multiple different functions, each with different “costs” interms, e.g., of response time, CPU utilization, memory usage, and/orother relevant constraints. The phone can then undertake a planningexercise, e.g., defining an N-ary tree composed of the various availableagents and functions, and navigating a path through the tree to discernhow to perform the desired combination of operations at the lowest cost.

Sometimes the planning exercise may not find a suitable solution, or mayfind its cost to be prohibitive. In such case the phone may decide notto undertake certain operations—at least not at the present instant. Thephone may do nothing further about such task, or it may try again amoment later, in case additional information has become available thatmakes a solution practical. Or it may simply refer to the data to thecloud—for processing by more capable cloud resources, or it may storethe input stimulus to revisit and possibly process later.

Much of the system's processing (e.g., image processing) may bespeculative in nature—tried in expectation that it might be useful inthe current context. In accordance with another aspect of the presenttechnology, such processes are throttled up or down in accordance withvarious factors. One factor is success. If a process seems to beproducing positive results, it can be allocated more resources (e.g.,memory, network bandwidth, etc.), and be permitted to continue intofurther stages of operation. If its results appear discouraging, it canbe allocated less resources—or stopped altogether. Another factor is theuser's interest in the outcome of a particular process, or lack thereof,which can similarly influence whether, and with what resources, aprocess is allowed to continue. (User interest may beexpress/explicit—e.g., by the user touching a location on the screen, orit may be inferred from the user's actions or context—e.g., by the usermoving the camera to re-position a particular subject in the center ofthe image frame. Lack of user interest may be similarly expressed by, orinferred from, the user's actions, or from the absence of such actions.)Still another factor is the importance of the process' result to anotherprocess that is being throttled up or down.

Once cognition has been achieved (e.g., once the subject of the imagehas been identified), the cell phone processor—or a cloud resource—maysuggest an appropriate response that should be provided to the user. Ifthe depicted subject is a barcode, one response may be indicated (e.g.,look up product information). If the depicted subject is a familymember, a different response may be indicated (e.g., post to an onlinephoto album). Sometimes, however, an appropriate response is notimmediately apparent. What if the depicted subject is a street scene, ora parking meter—what then? Again, collateral information sources, suchas context, and information from natural language processing, can beapplied to the problem to help determine appropriate responses.

The sensors of a smart phone are constantly presented with stimuli—soundto the microphone, light to the image sensor, motion to theaccelerometers and gyroscopes, magnetic fields to the magnetometer,ambient temperature to thermistors, etc., etc. Some of the stimulus maybe important. Much is noise, and is best ignored. The phone, of course,has a variety of limited resources, e.g., CPU, battery, wirelessbandwidth, dollar budget, etc.

Thus, in a further aspect, the present technology involves identifyingwhat of the barrage of data to process, and balancing data processingarrangements for the visual search with the constraints of the platform,and other needs of the system.

In still another aspect, the present technology involves presentation of“baubles” on a mobile device screen, e.g., in correspondence with visualobjects (or audible streams). User selection of a bauble (e.g., by atouch screen tap) leads to an experience related to the object. Thebaubles may evolve in clarity or size as the device progressivelyunderstands more, or obtains more information, about the object.

In early implementations, systems of the sort described will berelatively elementary, and not demonstrate much insight. However, byfeeding a trickle (or torrent) of data back to the cloud for archivingand analysis (together with information about user action based on suchdata), those early systems can establish the data foundation from whichtemplates and other training models can be built—enabling subsequentgenerations of such systems to be highly intuitive and responsive whenpresented with stimuli.

As will become evident, the present specification details a great numberof other inventive features and combinations as well.

While described primarily in the context of visual search, it should beunderstood that principles detailed herein are applicable in othercontexts, such as the processing of stimuli from other sensors, or fromcombinations of sensors. Many of the detailed principles have still muchbroader applicability.

Similarly, while the following description focuses on a few exemplaryembodiments, it should be understood that the inventive principles arenot limited to implementation in these particular forms. So, forexample, while details such as blackboard data structures, state machineconstructs, recognition agents, lazy execution, etc., etc., arespecifically noted, none (except as may be particularly specified byissued claims) is required.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an embodiment employing certain aspects of the presenttechnology, in an architectural view.

FIG. 2 is a diagram illustrating involvement of a local device withcloud processes.

FIG. 3 maps features of a cognitive process, with different aspects offunctionality—in terms of system modules and data structures.

FIG. 4 illustrates different levels of spatial organization andunderstanding.

FIGS. 5, 5A and 6 show data structures that can be used in makingcomposition of services decisions.

FIGS. 7 and 8 show aspects of planning models known from artificialintelligence, and employed in certain embodiments of the presenttechnology.

FIG. 9 identifies four levels of concurrent processing that may beperformed by the operating system.

FIG. 10 further details these four levels of processing for anillustrative implementation.

FIG. 11 shows certain aspects involved in discerning user intent.

FIG. 12 depicts a cyclical processing arrangement that can be used incertain implementations.

FIG. 13 is another view of the FIG. 12 arrangement.

FIG. 14 is a conceptual view depicting certain aspects of systemoperation.

FIGS. 15 and 16 illustrate data relating to recognition agents andresource tracking, respectively.

FIG. 17 shows a graphical target, which can be used to aid machineunderstanding of a viewing space.

FIG. 18 shows aspects of an audio-based implementation.

FIGS. 19 and 19A show a variety of possible user interface features.

FIG. 19B shows the lower, geolocation pane, of FIG. 19 in greaterdetail.

FIGS. 20A and 20B illustrate a method of object segmentation usingthresholded blobs.

FIGS. 21A, 21B and 22 show other exemplary user interface features.

FIGS. 23A and 23B show a radar feature in a user interface.

FIG. 24 serves to detail other user interface techniques.

FIGS. 25-30 illustrate features associated with declarativeconfiguration of sensor-related systems.

DETAILED DESCRIPTION

In many respects, the subject matter of this disclosure may be regardedas technologies useful in permitting users to interact with theirenvironments, using computer devices. This broad scope makes thedisclosed technology well suited for countless applications.

Due to the great range and variety of subject matter detailed in thisdisclosure, an orderly presentation is difficult to achieve. As will beevident, many of the topical sections presented below are both foundedon, and foundational to, other sections. Necessarily, then, the varioussections are presented in a somewhat arbitrary order. It should berecognized that both the general principles and the particular detailsfrom each section find application in other sections as well. To preventthe length of this disclosure from ballooning out of control(conciseness always being beneficial, especially in patentspecifications), the various permutations and combinations of thefeatures of the different sections are not exhaustively detailed. Theinventors intend to explicitly teach such combinations/permutations, butpracticality requires that the detailed synthesis be left to those whoultimately implement systems in accordance with such teachings.

It should also be noted that the presently-detailed technology buildson, and extends, technology disclosed in the earlier-cited patentapplications. The reader is thus directed to those documents, whichdetail arrangements in which applicants intend the present technology tobe applied, and that technically supplement the present disclosure.

Cognition, Disintermediated Search

Mobile devices, such as cell phones, are becoming cognition tools,rather than just communication tools. In one aspect, cognition may beregarded as activity that informs a person about the person'senvironment. Cognitive actions can include:

-   -   Perceiving features based on sensory input;    -   Perceiving forms (e.g., determining orchestrated structures);    -   Association, such as determining external structures and        relations;    -   Defining problems;    -   Defining problem solving status (e.g., it's text: what can I        do? A. Read it);    -   Determining solution options;    -   Initiating action and response;    -   Identification is generally the first, essential step in        determining an appropriate response.

Seeing and hearing mobile devices are tools that assist those processesinvolved in informing a person about their environment.

Mobile devices are proliferating at an amazing rate. Many countries(including Finland, Sweden, Norway, Russia, Italy, and the UnitedKingdom) reportedly have more cell phones than people. Accordingly tothe GSM Association, there are approximately 4 billion GSM and 3G phonescurrently in service. The International Telecommunications Unionestimates 4.9 billion mobile cellular subscriptions at the end of 2009.The upgrade cycle is so short that devices are replaced, on average,once every 24 months.

Accordingly, mobile devices have been the focus of tremendousinvestment. Industry giants such as Google, Microsoft, Apple and Nokia,have recognized that enormous markets hinge on extending thefunctionality of these devices, and have invested commensurately largesums in research and development. Given such widespread and intenseefforts, the failure of industry giants to develop the technologiesdetailed herein is testament to such technologies' inventiveness.

“Disintermediated search,” such as visual query, is believed to be oneof the most compelling applications for upcoming generations of mobiledevices.

In one aspect, disintermediated search may be regarded as search thatreduces (or even eliminates) the human's role in initiating the search.For example, a smart phone may always be analyzing the visualsurroundings, and offering interpretation and related informationwithout being expressly queried.

In another aspect, disintermediated search may be regarded as the nextstep beyond Google. Google built a monolithic, massive system toorganize all the textual information on the public web. But the visualworld is too big, and too complex, for even Google to master. Myriadparties are bound to be involved—each playing a specialized role, somelarger, some smaller. There will not be “one search engine to rule themall.” (Given the potential involvement of countless parties, perhaps analternative moniker would be “hyperintermediated search.”)

As will be apparent from the following discussion, the present inventorsbelieve that visual search, specifically, is extremely complicated incertain of its aspects, and requires an intimate device/cloudorchestration, supported by a highly interactive mobile screen userinterface, to yield a satisfactory experience. User guidance andinteraction is fundamental to the utility of the results—at leastinitially. On the local device, a key challenge is deploying scarceCPU/memory/channel/power resources against a dizzying array of demands.On the cloud side, auction-based service models are expected to emergeto drive evolution of the technology. Initially, disintermediated searchwill be commercialized in the form of closed systems, but to flourish,it will be via extensible, open platforms. Ultimately, the technologiesthat are most successful will be those that are deployed to provide thehighest value to the user.

Architectural View

FIG. 1 shows an embodiment employing certain principles of the presenttechnology, in an architectural view of an Intuitive Computing Platform,or ICP. (It should be recognized that the division of functionality intoblocks is somewhat arbitrary. Actual implementation may not follow theparticular organization depicted and described.)

The ICP Baubles & Spatial Model component handles tasks involving theviewing space, the display, and their relationships. Some of therelevant functions include pose estimation, tracking, andortho-rectified mapping in connection with overlaying baubles on avisual scene.

Baubles may be regarded, in one aspect, as augmented reality icons thatare displayed on the screen in association with features of capturedimagery. These can be interactive and user-tuned (i.e., differentbaubles may appear on the screens of different users, viewing theidentical scene).

In some arrangements, baubles appear to indicate a first glimmer ofrecognition by the system. When the system begins to discern thatthere's something of potential interest—a visual feature—at a locationon the display, it presents a bauble. As the system deduces more aboutthe feature, the size, form, color or brightness of the bauble maychange—making it more prominent, and/or more informative. If the usertaps the bauble—signifying interest in the visual feature, the system'sresource manager (e.g., the ICP State Machine) can devotedisproportionately more processing resources to analysis of that featureof the image than other regions. (Information about the user's tap alsois stored in a data store, in conjunction with information about thefeature or the bauble, so that the user's interest in that feature maybe recognized more quickly, or automatically, next time.)

When a bauble first appears, nothing may be known about the visualfeature except that it seems to constitute a visually discrete entity,e.g., a brightish spot, or something with an edge contour. At this levelof understanding, a generic bauble (perhaps termed a “proto-bauble”) canbe displayed, such as a small star or circle. As more information isdeduced about the feature (it appears to be a face, or bar code, orleaf), then a bauble graphic that reflects that increased understandingcan be displayed.

Baubles can be commercial in nature. In some environments the displayscreen could be overrun with different baubles, vying for the user'sattention. To address this, there can be a user-settable control—avisual verbosity control—that throttles how much information ispresented on the screen. In addition, or alternatively, a control can beprovided that allows the user to establish a maximum ratio of commercialbaubles vs. non-commercial baubles. (As with Google, collection of rawdata from the system may prove more valuable in the long term thanpresenting advertisements to users.)

Desirably, the baubles selected for display are those that serve thehighest value to the user, based on various dimensions of currentcontext. In some cases—both commercial and non-commercial—baubles may beselected based on auction processes conducted in the cloud. The finalroster of displayed baubles can be influenced by the user. Those withwhich the user interacts become evident favorites and are more likelydisplayed in the future; those that the user repeatedly ignores ordismisses may not be shown again.

Another GUI control can be provided to indicate the user's currentinterest (e.g., sightseeing, shopping, hiking, social, navigating,eating, etc.), and the presentation of baubles can be tuned accordingly.

In some respects, the analogy of an old car radio—with a volume knob onthe left and a tuning knob on the right—is apt. The volume knobcorresponds to a user-settable control over screen busyness (visualverbosity). The tuning knob corresponds to sensors, stored data, anduser input that, individually or in conjunction, indicate what type ofcontent is presently relevant to the user, e.g., the user's likelyintent.

The illustrated ICP Baubles & Spatial Model component may borrow from,or be built based on, existing software tools that serve relatedfunctions. One is the ARToolKit—a freely available set of softwareresulting from research at the Human Interface Technology Lab at theUniversity of Washington (hit121 dot>Washington<dot>edu/artoolkit/), nowbeing further developed by AR Toolworks, Inc., of Seattle(artoolworks<dot>com). Another related set of tools is MV Tools—apopular library of machine vision functions.

FIG. 1 shows just a few recognition agents (RAs); there may be dozens orhundreds. RAs include the components that perform feature and formextraction, and assist in association and identification, based onsensor data (e.g., pixels), and/or derivatives (e.g., “keyvector” data,c.f., US20100048242, WO10022185). They generally help recognize, andextract meaning from, available information. In one aspect, some RAs maybe analogized to specialized search engines. One may search for barcodes; one may search for faces, etc. (RAs can be of other types aswell, e.g., processing audio information, providing GPS and magnetometerdata, etc., in service of different processing tasks.)

RAs can execute locally, remotely, or both—based on the needs of thesession and the environment. They may be remotely loaded and operated,per device/cloud negotiated business rules. RAs commonly take, as input,keyvector data from a shared data structure, the ICP blackboard(discussed below). They may provide elemental services that arecomposited by the ICP state machine in accordance with a solution tree.

As with baubles, there may be an aspect of competition involving RAs.That is, overlapping functionality may be offered by several differentRAs from several different providers. The choice of which RA to use on aparticular device in a particular context can be a function of userselection, third party reviews, cost, system constraints, re-usabilityof output data, and/or other criteria. Eventually, a Darwinian winnowingmay occur, with those RAs that best meet users' needs becomingprevalent.

A smart phone vendor may initially provide the phone with a default setof RAs. Some vendors may maintain control of RA selection—a walledgarden approach, while others may encourage user discovery of differentRAs. Online marketplaces such as the Apple App Store may evolve to servethe RA market. Packages of RAs serving different customer groups andneeds may emerge, e.g., some to aid people with limited vision (e.g.,loaded with vision-aiding RAs, such as text-to-speech recognition), somecatering to those who desire the simplest user interfaces (e.g., largebutton controls, non-jargon legends); some catering to outdoorenthusiasts (e.g., including a birdsong identification RA, a tree leafidentification RA); some catering to world travelers (e.g., includinglanguage translation functions, and location-based traveler services),etc. The system may provide a menu by which a user can cause the deviceto load different such sets of RAs at different times.

Some, or all, of the RAs may push functionality to the cloud, dependingon circumstance. For example, if a fast data connection to the cloud isavailable, and the device battery is nearing exhaustion (or if the useris playing a game—consuming most of the device's CPU/GPU resources),then the local RA may just do a small fraction of the task locally(e.g., administration), and ship the rest to a cloud counterpart, forexecution there.

As detailed elsewhere, the processor time and other resources availableto RAs can be controlled in dynamic fashion—allocating more resources tothose RAs that seem to merit it. A dispatcher component of the ICP statemachine can attend to such oversight. The ICP state machine can alsomanage the division of RA operation between local RA components andcloud counterparts.

The ICP state machine can employ aspects modeled from the Android opensource operating system (e.g.,developer<dot>android<dot>com/guide/topics/fundamentals.html), as wellas from the iPhone and Symbian SDKs.

To the right in FIG. 1 is the Cloud & Business Rules Component, whichserves as an interface to cloud-relating processes. It can also performadministration for cloud auctions—determining which of plural cloudservice providers performs certain tasks. It communicates to the cloudover a service provider interface (SPI), which can utilize essentiallyany communications channel and protocol.

Although the particular rules will be different, exemplary rules-basedsystems that can be used as models for this aspect of the architectureinclude the Movielabs Content Rules and Rights arrangement (e.g.,movielabs<dot>com/CRR/), and the CNRI Handle System (e.g.,handle<dot>net/).

To the left is a context engine which provides, and processes, contextinformation used by the system (e.g., What is the current location? Whatactions has the user performed in the past minute? In the past hour?etc.). The context component can link to remote data across aninterface. The remote data can comprise any external information, e.g.,concerning activities, peers, social networks, consumed content,geography—anything that may relate the present user to others—such as asimilar vacation destination. (If the device includes a musicrecognition agent, it may consult playlists of the user's Facebookfriends. It may use this information to refine a model of music that theuser listens to—also considering, e.g., knowledge about what onlineradio stations the user is subscribed to, etc.)

The context engine, and the cloud & business rules components, can havevestigial cloud-side counterparts. That is, this functionality can bedistributed, with part local, and a counterpart in the cloud.Cloud-based interactions can utilize many of the tools and softwarealready published for related cloud computing by Google's App Engine(e.g., code<dot>Google<dot>com/appengine/) and Amazon's Elastic ComputeCloud (e.g., aws<dot>amazon<dot>com/ec2/).

At the bottom in FIG. 1 is the Blackboard and Clustering Engine.

The blackboard can serve various functions, including as a shared datarepository, and as a means for interprocess communication—allowingmultiple recognition agents to observe and contribute feature objects(e.g., keyvectors), and collaborate. It may serve as a data model forthe system, e.g., maintaining a visual representation to aid in featureextraction and association across multiple recognition agents, providingcaching and support for temporal feature/form extraction, and providingmemory management and trash services. It can also serve as a featureclass factory, and provide feature object instantiation (creation anddestruction, access control, notification, serialization in the form ofkeyvectors, etc.).

Blackboard functionality can utilize the open source blackboard softwareGBBopen (gbbopen<dot>org). Another open source implementation that runson the Java Virtual Machine (and supports scripting in JavaScript) isthe Blackboard Event Processor(code<dot>Google<dot>com/p/blackboardeventprocessor/).

The blackboard construct was popularized by Daniel Corkill. See, e.g.,Corkill, Collaborating Software—Blackboard and Multi-Agent Systems & theFuture, Proceedings of the International Lisp Conference, 2003. However,implementation of the present technology does not require any particularform of the concept.

The Clustering Engine groups items of content data (e.g., pixels)together, e.g., in keyvectors. Keyvectors can, in one aspect, be roughlyanalogized as audio-visual counterpart to text keywords—a grouping ofelements that are input to a process to obtain related results.

Clustering can be performed by low-level processes that generate newfeatures from image data—features that can be represented as lists ofpoints, vectors, image regions, etc. (Recognition operations commonlylook for clusters of related features, as they potentially representobjects of interest.) These features can be posted to the blackboard.(Higher level processes—which may form part of recognition agents—canalso generate new features or objects of interest, and post them to theblackboard as well.)

Again, the earlier-referenced ARToolKit can provide a basis for certainof this functionality.

Aspects of the foregoing are further detailed in the following and othersections of this specification.

Local Device & Cloud Processing

As conceptually represented by FIG. 2, disintermediated search shouldrely on strengths/attributes of the local device and of the cloud. (Thecloud “pipe” also factors into the mix, e.g., by constraints includingbandwidth and cost.)

The particular distribution of functionality between the local deviceand the cloud varies from implementation to implementation. In oneparticular implementation it is divided as follows:

Local Functionality:

-   -   Context:        -   User identity, preferences, history        -   Context Metadata Processing (e.g., where am I? what            direction am I pointing?)    -   UI:        -   On screen rendering & feedback (touch, buttons, audible,            proximity, etc.)    -   General Orientation:        -   Global sampling; categorization without much parsing        -   Data alignment and feature extraction        -   Enumerated patchworks of features        -   Interframe collections; sequence of temporal features    -   Cloud Session Management:        -   Registration, association & duplex session operations with            recognition agents    -   Recognition agent management:        -   Akin to DLLs with specific functionality—recognizing            specific identities and forms        -   Resource state and detection state scalability        -   Composition of services provided by recognition agents        -   Development and licensing platform

Cloud roles may include, e.g:

-   -   Communicate with enrolled cloud-side services    -   Manage and execute auctions for services (and/or audit auctions        on the device)    -   Provide/support identity of users and objects, e.g., by        providing services associated with the seven laws of identity        (c.f., Microsoft's Kim Cameron):        -   User control and consent. Technical identity systems must            only reveal information identifying a user with the user's            consent.        -   Minimal disclosure for a constrained use. The solution that            discloses the least amount of identifying information and            best limits its use is the most stable long-term solution.        -   Justifiable parties. Digital identity systems must be            designed so the disclosure of identifying information is            limited to parties having a necessary and justifiable place            in a given identity relationship.        -   Directed identity. A universal identity system must support            both “omnidirectional” identifiers for use by public            entities and “unidirectional” identifiers for use by private            entities, thus facilitating discovery while preventing            unnecessary release of correlation handles.        -   Pluralism of operators and technologies. A universal            identity system must channel and enable the inter-working of            multiple identity technologies run by multiple identity            providers.        -   Human integration. The universal identity metasystem must            define the human user to be a component of the distributed            system integrated through unambiguous human/machine            communication mechanisms, offering protection against            identity attacks.        -   Consistent experience across contexts. The unifying identity            metasystem must guarantee its users a simple, consistent            experience while enabling separation of contexts through            multiple operators and technologies.    -   Create and enforce construct of domain        -   Billing, geography, device, content    -   Execute and control recognition agents within user initiated        sessions    -   Manage remote recognition agents (e.g., provisioning,        authentication, revocation, etc.)    -   Attend to business rules and session management, etc.

The Cloud not only facilitates disintermediated search, but often is thedestination of the search as well (except in cases such as OCR, whereresults generally can be provided based on sensor data alone);

The presently-detailed technologies draw inspiration from diversesources, including:

Biological: Analogies to Human Visual System & higher level cognitionmodels

Signal Processing: Sensor Fusion

Computer Vision: Image processing Operations (spatial & frequencydomain)

Computer Science: Composition of Services & Resource Management,Parallel Computing

Robotics: Software models for autonomous interaction (PLAN, Gazebo,etc.)

AI: Match/Deliberate/Execute Models, Blackboard, Planning Models, etc.

Economics: Auction Models (Second Price Wins . . . )

DRM: Rights Expression Languages & Business Rule engines

Human Factors: UI, Augmented Reality,

Mobile Value Chain Structure: Stakeholders, Business Models, Policy,etc.

Behavioral Science: Social Networks, Crowdsourcing/Folksonomies,

Sensor Design: Magnetometers, Proximity, GPS, Audio, Optical (ExtendedDepth of Field, etc.)

FIG. 3 maps the various features of an illustrative cognitive process,with different aspects of functionality—in terms of system modules anddata structures. Thus, for example, an Intuitive Computing Platform(ICP) Context Engine applies cognitive processes of association, problemsolving status, determining solutions, initiating actions/responses, andmanagement, to the context aspect of the system. In other words, the ICPContext Engine attempts to determine the user's intent based on history,etc., and use such information to inform aspects of system operation.Likewise, the ICP Baubles & Spatial Model components serve many of thesame processes, in connection with presenting information to the user,and receiving input from the user.

The ICP Blackboard and keyvectors are data structures used, among otherpurposes, in association with orientation aspects of the system.

ICP State Machine & Recognition Agent Management, in conjunction withrecognition agents, attend to recognition processes, and composition ofservices associated with recognition. The state machine is typically areal-time operating system. (Such processes also involve, e.g., the ICPBlackboard and keyvectors.)

Cloud Management & Business Rules deals with cloud registration,association, and session operations—providing an interface betweenrecognition agents and other system components, and the cloud.

Local Functionality to Support Baubles

Some of the functions provided by one or more of the software componentsrelating to baubles can include the following:

-   -   Understand the user's profile, their general interests, their        current specific interests within their current context.    -   Respond to user inputs.    -   Spatially parse and “object-ify” overlapping scene regions of        streaming frames using selected modules of a global image        processing library        -   Attach hierarchical layers of symbols (pixel analysis            results, IDs, attributes, etc.) to proto-regions; package up            as “key vectors” of proto-queries.    -   Based on user-set visual verbosity levels and global scene        understanding, set up bauble primitive display        functions/orthography.    -   Route keyvectors to appropriate local/cloud addresses        -   With attached “full context” metadata from top listed            bullet.        -   If local: process the keyvectors and produce query results.    -   Collect keyvector query results and enliven/blit appropriate        baubles to user screen        -   Baubles can be either “complete and fully actionable,” or            illustrate “interim states” and hence expect user            interaction for deeper query drilling or query refinement.

Intuitive Computing Platform (ICP) Baubles

Competition in the cloud for providing services and high value baubleresults should drive excellence and business success for suppliers.Establishing a cloud auction place, with baseline quality non-commercialservices, may help drive this market.

Users want (and should demand) the highest quality and most relevantbaubles, with commercial intrusion tuned as a function of theirintentions and actual queries.

On the other side, buyers of screen real estate may be split into twoclasses: those willing to provide non-commercial baubles and sessions(e.g., with the goal of gaining a customer for branding), and thosewanting to “qualify” the screen real estate (e.g., in terms of thedemographics of the user(s) who will see it), and simply bid on thecommercial opportunities it represents.

Google, of course, has built a huge business on monetizing its “keyword, to auction process, to sponsored hyperlink presentation”arrangements. However, for visual search, it seems unlikely that asingle entity will similarly dominate all aspects of the process.Rather, it seems probable that a middle layer of companies will assistin the user query/screen real estate buyer-matchmaking.

The user interface may include a control by which the user can dismissbaubles that are of no interest—removing them from the screen (andterminating any on-going recognition agent process devoted to developingfurther information relating to that visual feature). Information aboutbaubles that are dismissed can be logged in a data store, and used toaugment the user's profile information. If the user dismisses baublesfor Starbucks coffee shops and independent coffee shops, the system maycome to infer a lack of interest by the user in all coffee shops. If theuser dismisses baubles only for Starbucks coffee shops, then a morenarrow lack of user interest can be discerned. Future displays ofbaubles can consult the data store; baubles earlier dismissed (orrepeatedly dismissed) may not normally be displayed again.

Similarly, if the user taps on a bauble—indicating interest—then thattype or class of bauble (e.g., Starbucks, or coffee shops) can be givena higher score in the future, in evaluating which baubles (among manycandidates) to display.

Historical information about user interaction with baubles can be usedin conjunction with current context information. For example, if theuser dismisses baubles relating to coffee shops in the afternoons, butnot in the mornings, then the system may continue to presentcoffee-related baubles in the morning.

The innate complexity of the visual query problem implies that manybaubles will be of an interim, or proto-bauble class—inviting andguiding the user to provide human-level filtering, interaction, guidanceand navigation deeper into the query process. The progression of baubledisplays on a scene can thus be a function of real-time human input, aswell as other factors.

When a user taps, or otherwise expresses interest in, a bauble, thisaction usually initiates a session relating to the subject matter of thebauble. The details of the session will depend on the particular bauble.Some sessions may be commercial in nature (e.g., tapping on a Starbucksbauble may yield an electronic coupon for a dollar off a Starbucksproduct). Others may be informational (e.g., tapping on a baubleassociated with a statue may lead to presentation of a Wikipedia entryabout the statue, or the sculptor). A bauble indicating recognition of aface in a captured image might lead to a variety of operations (e.g.,presenting a profile of the person from a social network, such asLinkedIn; posting a face-annotated copy of the picture to the Facebookpage of the recognized person or of the user, etc.). Sometimes tapping abauble summons a menu of several operations, from which the user canselect a desired action.

Tapping a bauble represents a victory of sorts for that bauble, overothers. If the tapped bauble is commercial in nature, that bauble haswon a competition for the user's attention, and for temporary usage ofreal estate on the viewer's screen. In some instances, an associatedpayment may be made—perhaps to the user, perhaps to another party (e.g.,an entity that secured the “win” for a customer).

A tapped bauble also represents a vote of preference—a possibleDarwinian nod to that bauble over others. In addition to influencingselection of baubles for display to the present user in the future, suchaffirmation can also influence the selection of baubles for display toother users. This, hopefully, will lead bauble providers into a virtuouscircle toward user-serving excellence. (How many current televisioncommercials would survive if only user favorites gained ongoingairtime?)

As indicated, a given image scene may provide opportunities for displayof many baubles—often many more baubles that the screen can usefullycontain. The process of narrowing this universe of possibilities down toa manageable set can begin with the user.

A variety of different user input can be employed, starting with averbosity control as indicated earlier—simply setting a baseline for howbusily the user wants the screen to be overlaid with baubles. Othercontrols may indicate topical preferences, and a specified mix ofcommercial to non-commercial.

Another dimension of control is the user's real-time expression ofinterest in particular areas of the screen, e.g., indicating featuresabout which the user wants to learn more, or otherwise interact. Thisinterest can be indicated by tapping on proto-baubles overlaid on suchfeatures, although proto-baubles are not required (e.g., a user maysimply tap an undifferentiated area of the screen to focus processorattention to that portion of the image frame).

Additional user input is contextual—including the many varieties ofinformation detailed elsewhere (e.g., computing context, physicalcontext, user context, physical context, temporal context and historicalcontext).

External data that feeds into the bauble selection process can includeinformation relating to third party interactions—what baubles did otherschoose to interact with? The weight given this factor can depend on adistance measure between the other user(s) and the present user, and adistance between their context and the present context. For example,bauble preferences expressed by actions of social friends of the presentuser, in similar contextual circumstances, can be given much greaterweight than actions of strangers in different circumstances.

Another external factor can be commercial considerations, e.g., how much(and possibly to whom) a third party is willing to pay in order tobriefly lease a bit of the user's screen real estate. As noted, suchissues can factor into a cloud-based auction arrangement. The auctioncan also take into account the popularity of particular baubles withother users. In implementing this aspect of the process, reference maybe made to the Google technology for auctioning online advertising realestate (see, e.g., Levy, Secret of Googlenomics: Data-Fueled RecipeBrews Profitability, Wired Magazine, May 22, 2009)—a variant of ageneralized second-price auction. Applicants detailed cloud-basedauction arrangements in published PCT application WO2010022185.

(Briefly, the assumption of such cloud-based models is that they areakin to advertising models based on click thru rates (CTR): entitieswill pay varying amounts (monetary and/or subsidized services) to ensurethat their service is used, and/or that their baubles appear on users'screens. Desirably, there is a dynamic marketplace for recognitionservices offered by commercial and non-commercial recognition agents(e.g., a logo recognition agent that already has Starbucks logospre-cached). Lessons can also be gained from search-informedadvertising—the balance is providing user value while profiting ontraffic.)

Generally, the challenges in these auctions are not in conduct of theauction, but rather suitably addressing the number of variablesinvolved. These include:

-   -   User profile (e.g., based on what is known—such as by cookies in        the browser world—how much does a vendor want to expend to place        a bauble?)    -   Cost (what the bandwidth, computational and opportunity costs?);        and    -   Device capabilities (both in static terms, such as hardware        provision—flash? GPU?, etc., and also in terms of dynamic state,        such as the channel bandwidth at the user's current location,        the device's power state, memory usage, etc.)

(In some implementations, bauble promoters may try harder to placebaubles on screens of well-heeled users, as indicated by the type ofdevice they are using. A user with the latest, most expensive type ofdevice, or using an expensive data service, may merit more commercialattention than a user with an antiquated device, or the trailing edgedata service. Other profile data exposed by the user, or inferable fromcircumstances, can similarly be used by third parties in deciding whichscreens are the best targets for their baubles.)

In one particular implementation, a few baubles (e.g., 1-8) may beallocated to commercial promotions (e.g., as determined by a Google-likeauction procedure, and subject to user tuning of commercial vs.non-commercial baubles), and others may be selected based onnon-commercial factors, such as noted earlier. These latter baubles maybe chosen in rule-based fashion, e.g., applying an algorithm thatweights different factors noted earlier to obtain a score for eachbauble. The competing scores are then ranked, and the highest-scoring Nbaubles (where N may be user-set using the verbosity control) arepresented on the screen.

In another implementation, there is no a priori allocation forcommercial baubles. Instead, these are scored in a manner akin to thenon-commercial baubles (typically using different criteria, but scaledto a similar range of scores). The top-scoring N baubles are thenpresented—which may be all commercial, all non-commercial, or a mix.

In still another implementation, the mix of commercial to non-commercialbaubles is a function of the user's subscription service. Users at anentry level, paying an introductory rate, are presented commercialbaubles that are large in size and/or number. Users paying a serviceprovider for premium services are presented smaller and/or fewercommercial baubles, or are given latitude to set their own parametersabout display of commercial baubles.

The graphical indicia representing a bauble can be visually tailored toindicate its feature association, and may include animated elements toattract the user's attention. The bauble provider may provide the systemwith indicia in a range of sizes, allowing the system to increase thebauble size—and resolution—if the user zooms into that area of thedisplayed imagery, or otherwise expresses potential interest in suchbauble. In some instances the system must act as cop—deciding not topresent a proffered bauble, e.g., because its size exceeds dimensionsestablished by stored rules, its appearance is deemed salacious, etc.(The system may automatically scale baubles down to a suitable size, andsubstitute generic indicia—such as a star—for indicia that areunsuitable or otherwise unavailable.)

Baubles can be presented other than in connection with visual featuresdiscerned from the imagery. For example, a bauble may be presented toindicate that the device knows its geolocation, or that the device knowsthe identity of its user. Various operational feedback can thus beprovided to the user—regardless of image content. Some image feedbackmay also be provided via baubles—apart from particular featureidentification, e.g., that the captured imagery meets baseline qualitystandards such as focus or contrast.

Each bauble can comprise a bit mapped representation, or it can bedefined in terms of a collection of graphical primitives. Typically, thebauble indicia is defined in plan view. The spatial model component ofthe software can attend to mapping its projection onto the screen inaccordance with discerned surfaces within the captured imagery, e.g.,seemingly inclining and perhaps perspectively warping a baubleassociated with an obliquely-viewed storefront. Such issues arediscussed further in the following section.

Spatial Model/Engine

Satisfactory projection and display of the 3D world onto a 2D screen canbe important in establishing a pleasing user experience. Accordingly,the preferred system includes software components (variously termed,e.g., spatial model or a spatial engine) to serve such purposes.

Rendering of the 3D world in 2D starts by understanding something aboutthe 3D world. From a bare frame of pixels—lacking any geolocation dataor other spatial understanding—where to begin? How to discern objects,and categorize? How to track movement of the image scene, so thatbaubles can be repositioned accordingly? Fortunately, such issues havebeen confronted many times in many situations. Machine vision and videomotion encoding are two fields, among many, that provide useful priorart with which the artisan is presumed to be familiar, and from whichthe artisan can draw in connection with the present application.

By way of first principles:

The camera and the displayed screen are classic 2D spatial structures

The camera functions through spatial projections of the 3D world onto a2D plane

Baubles and proto-baubles are “objectified” within a spatial framework.

Below follows a proposal to codify spatial understanding as anorthogonal process stream, as well as a context item and an attributeitem. It utilizes the construct of three “spacelevels”—stages of spatialunderstanding.

Spacelevel 1 comprises basic scene analysis and parsing. Pixels areclumped into initial groupings. There is some basic understanding of thecaptured scene real estate, as well as display screen real estate. Thereis also some rudimentary knowledge about the flow of scene real estateacross frames.

Geometrically, Spacelevel 1 lives in the context of a simple 2D plane.Spacelevel 1 operations include generating lists of 2D objects discernedfrom pixel data. The elemental operations performed by the OpenCV visionlibrary (discussed below) fall in this realm of analysis. The smartphone's local software may be fluent in dealing with Spacelevel 1operations, and rich lists of 2D objects may be locally produced.

Spacelevel 2 is transitional—making some sense of the Spacelevel 1 2Dprimitives, but not yet to the full 3D understanding of Spacelevel 3.This level of analysis includes tasks seeking to relate differentSpacelevel 1 primitives—discerning how objects relate in a 2D context,and looking for clues to 3D understanding. Included are operations suchas identifying groups of objects (e.g., different edges forming anoutline defining a shape), noting patterns—such as objects along a line,and discerning “world spatial clues” such as vanishing points, horizons,and notions of “up/down.” Notions of “closer/further” may also beuncovered. (E.g., a face has generally known dimensions. If a set ofelemental features seems to likely represent a face, and the set is only40 pixels tall in a scene that is 480 pixels tall, then a “further”attribute may be gathered—in contrast to a facial collection of pixelsthat is 400 pixels tall.)

The cacophony of Spacelevel 1 primitives is distilled/composited intoshorter, more meaningful lists of object-related entities.

Spacelevel 2 may impose a GIS-like organization onto scene and scenesequences, e.g., assigning each identified clump, object, or region ofinterest, its own logical data layer—possibly with overlapping areas.Each layer may have an associated store of metadata. In this level,object continuity—frame-to-frame, can be discerned.

Geometrically, Spacelevel 2 acknowledges that the captured pixel data isa camera's projection of a 3D world onto a 2D image frame. Theprimitives and objects earlier discerned are not taken to be a fullcharacterization of reality, but rather one view. Objects are regardedin the context of the camera lens from which they are viewed. The lensposition establishes a perspective from which the pixel data should beunderstood.

Spacelevel 2 operations typically tend to rely more on cloud processingthan Spacelevel 1 operations.

In the exemplary embodiment, the Spatial Model components of thesoftware are general purpose—distilling pixel data into more usefulform. The different recognition agents can then draw from this commonpool of distilled data in performing their respective tasks, rather thaneach doing their own version of such processing. A line must be drawn,however, in deciding which operations are of such general utility thatthey are performed in this common fashion as a matter of course, andwhich operations should be relegated to individual recognitionagents—performed only as needed. (Their results may nonetheless beshared, e.g., by the blackboard.) The line can be drawn arbitrarily; thedesigner has freedom to decide which operations fall on which side ofthe line. Sometimes the line may shift dynamically during a phone'soperation, e.g., if a recognition agent makes a request for furthercommon services support.

Spacelevel 3 operations are based in 3D. Whether or not the data revealsthe full 3D relationships (it generally will not), the analyses arebased on the premise that the pixels represent a 3D world. Suchunderstanding is useful—even integral—to certain object recognitionprocesses.

Spacelevel 3 thus builds on the previous levels of understanding,extending out to world correlation. The user is understood to be anobserver within a world model with a given projection and spacetimetrajectory. Transformation equations mapping scene-to-world, andworld-to-scene, can be applied so that the system understands both whereit is in space, and where objects are in space, and has some frameworkfor how things relate. These phases of analysis draw from work in thegaming industry, and augmented reality engines.

Unlike operations associated with Spacelevel 1 (and some with Spacelevel2), operations associated with Spacelevel 3 are generally so specializedthat they are not routinely performed on incoming data (at least notwith current technology). Rather, these tasks are left to particularrecognition tasks that may require particular 3D information.

Some recognition agents may construct a virtual model of the user'senvironment—and populate the model with sensed objects in their 3Dcontext. A vehicle driving monitor, for example, may look out thewindshield of the user's car—noting items and actions relevant totraffic safety. It may maintain a 3D model of the traffic environment,and actions within it. It may take note of the user's wife (identifiedby another software agent, which posted the identification to theblackboard) driving her red Subaru through a red light—in view of theuser. 3D modeling to support such functionality is certainly possible,but is not the sort of operation that would be performed routinely bythe phone's general services.

Some of these aspects are shown in FIG. 4, which conceptuallyillustrates the increasing sophistication of spatial understanding fromSpacelevel 1, to 2, to 3.

In an illustrative application, different software components areresponsible for discerning the different types of information associatedwith the different Spacelevels. A clumping engine, for example, is usedin generating some of the Spacelevel 1 understanding.

Clumping refers to the process for identifying a group of (generallycontiguous) pixels as related. This relation can be, e.g., similarity incolor or texture. Or it can be similarity in flow (e.g., a similarpattern of facial pixels shifting across a static background from frameto frame).

In one arrangement, after the system has identified a clump of pixels,it assigns symbology (e.g., as simple as an ID number) to be associatedwith the clump. This is useful in connection with further management andanalysis of the clump (and otherwise as well, e.g., in connection withlinked data arrangements). A proto-bauble may be assigned to the clump,and tracked by reference to the identifying symbol. Informationresulting from parsing and orientation operations performed by thesystem, relating the clump's position to that of the camera in 2D and3D, may be organized by reference to the clump's symbol. Similarly, dataresulting from image processing operations associated with a clump canbe identified by reference to the clump's symbol. Likewise, user tapsmay be logged in association with the symbol. This use of the symbol asa handle by which clump-related information can be stored and managedcan extend to cloud-based processes relating to the clump, the evolutionof the bauble associated with a clump, all the way through fullrecognition of the clump-object and responses based thereon. (Moredetailed naming constructs, e.g., including session IDs, are introducedbelow.)

These spatial understanding components can operate in parallel withother system software components, e.g., maintaining common/globalspatial understanding, and setting up a spatial framework that agentsand objects can utilize. Such operation can include posting currentinformation about the spatial environment to a sharable data structure(e.g., blackboard) to which recognition agents can refer to helpunderstand what they are looking at, and which the graphics system canconsult in deciding how to paint baubles on the current scenery.Different objects and agents can set up spacelevel fields and attributeitems associated with the three levels.

Through successive generations of these systems, the spatialunderstanding components are expected to become an almost reflexive,rote capability of the devices.

Intuitive Computing Platform (ICP) State Machine—Composition ofServices; Service Oriented Computing; Recognition Agents

As noted earlier, the ICP state machine can comprise, in essence, a realtime operating system. It can attend to traditional tasks such asscheduling, multitasking, error recovery, resource management, messagingand security, and some others that are more particular to the currentapplications. These additional tasks may include providing audit trailfunctionality, attending to secure session management, and determiningcomposition of services.

The audit trail functionality provides assurance to commercial entitiesthat the baubles they paid to sponsor were, in fact, presented to theuser.

Secure session management involves establishing and maintainingconnections with cloud services and other devices that are robust fromeavesdropping, etc. (e.g., by encryption).

Composition of services refers to the selection of operations forperforming certain functions (and related orchestration/choreography ofthese component operations). A dispatch process can be involved in theseaspects of the state machine's operation, e.g., matching up resourceswith applications.

Certain high level functions may be implemented using data fromdifferent combinations of various lower level operations. The selectionof which functions to utilize, and when, can be based on a number offactors. One is what other operations are already underway orcompleted—the results of which may also serve the present need.

To illustrate, barcode localization may normally rely on calculation oflocalized horizontal contrast, and calculation of localized verticalcontrast, and comparison of such contrast data. However, if 2D EFT datafor 16×16 pixel tiles across the image is already available from anotherprocess, then this information might be used to locate candidate barcodeareas instead.

Similarly, a function may need information about locations of long edgesin an image, and an operation dedicated to producing long edge datacould be launched. However, another process may have already identifiededges of various lengths in the frame, and these existing results maysimply be filtered to identify the long edges, and re-used.

Another example is Hough transform-based feature recognition. The OpenCVvision library indicates that this function desirably uses thinned-edgeimage data as input data. It further recommends generating thethinned-edge image data by applying a Canny operation to edge data. Theedge data, in turn, is commonly generated by applying a Sobel filter tothe image data So, a “by the book” implementation of a Hough procedurewould start with a Sobel filter, followed by a Canny operation, and theninvoke the Hough method.

But edges can be determined by methods other than a Sobel filter. Andthinned edges can be determined by methods other than Canny. If thesystem already has edge data—albeit generated by a method other than aSobel filter, this edge data may be used. Similarly, if another processhas already produced reformed edge data—even if not by a Cannyoperation, this reformed edge data may be used.

In one particular implementation, the system (e.g., a dispatch process)can refer to a data structure having information that establishes roughdegrees of functional correspondence between different types ofkeyvectors. Keyvector edge data produced by Canny may be indicated tohave a high degree of functional correspondence with edge data producedby the Infinite Symmetric Exponential Filter technique, and a somewhatlesser correspondence with edge data discerned by the Marr-Hildrethprocedure. Corners detected by a Harris operator may be interchangeablewith corners detected by the Shi and Tomasi method. Etc.

This data structure can comprise one large table, or it can be brokendown into several tables—each specialized to a particular type ofoperation. FIG. 5, for example, schematically shows part of a tableassociated with edge finding—indicating a degree of correspondence(scaled to 100).

A particular high level function (e.g., barcode decoding) may call fordata generated by a particular process, such as a Canny edge filter. ACanny filter function may be available in a library of softwareprocessing algorithms available to the system, but before invoking thatoperation the system may consult the data structure of FIG. 5 to see ifsuitable alternative data is already available, or in-process (assumingthe preferred Canny data is not already available).

The check begins by finding the row having the nominally desiredfunction in the left-most column. The procedure then scans across thatrow for the highest value. In the case of Canny, the highest value is95, for Infinite Symmetric Exponential Filter. The system can check theshared data structure (e.g., blackboard) to determine whether such datais available for the subject image frame (or a suitable substitute). Iffound, it may be used in lieu of the nominally-specified Canny data, andthe barcode decoding operation can continue on that basis. If none isfound, the state machine process continues—looking for next-highestvalue(s) (e.g., 90 for Marr-Hildreth). Again, the system checks whetherany data of this type is available. The process proceeds until all ofthe alternatives in the table are exhausted.

In a presently preferred embodiment, this checking is undertaken by thedispatch process. In such embodiment, most recognition processes areperformed as cascaded sequences of operations—each with specifiedinputs. Use of a dispatch process allows the attendant composition ofservices decision-making to be centralized. This also allows theoperational software components to be focused on image processing,rather than also being involved, e.g., with checking tables for suitableinput resources and maintaining awareness of operations of otherprocesses—burdens that would make such components more complex anddifficult to maintain.

In some arrangements, a threshold is specified—by the barcode decodingfunction, or by the system globally, indicating a minimum correspondencevalue that is acceptable for data substitution, e.g., 75. In such case,the just-described process would not consider data from Sobel and Kirchfilters—since their degree of correspondence with the Canny filter isonly 70.

Although other implementations may be different, note that the table ofFIG. 5 is not symmetrical. For example, if Canny is desired, Sobel hasan indicated correspondence of only 70. But if Sobel is desired, Cannyhas an indicated correspondence of 90. Thus, Canny may be substitutedfor Sobel, but not vice versa, if a threshold of 75 is set.

The table of FIG. 5 is general purpose. For some particularapplications, however, it may not be suitable. A function, for example,may require edges be found with Canny (preferred), or Kirch orLaplacian. Due to the nature of the function, no other edge finder maybe satisfactory.

The system can allow particular functions to provide their owncorrespondence tables for one or more operations—pre-empting applicationof the general purpose table(s). The existence of specializedcorrespondence tables for a function can be indicated by a flag bitassociated with the function, or otherwise. In the example just given, aflag bit may indicate that the table of FIG. 5A should be used instead.This table comprises just a single row—for the Canny operation that isnominally specified for use in the function. And it has just twocolumns—for Infinite Symmetric Exponential Filter and Laplacian. (Noother data is suitable.) The correspondence values (i.e., 95, 80) may beomitted—so that the table can comprise a simple list of alternativeprocesses.

To facilitate finding substitutable data in the shared data structure, anaming convention can be used indicating what information a particularkeyvector contains. Such a naming convention can indicate a class offunction (e.g., edge finding), a particular species of function (e.g.,Canny), the image frame(s) on which the data is based, and any otherparameters particular to the data (e.g., the size of a kernel for theCanny filter). This information can be represented in various ways, suchas literally, by abbreviation, by one or more index values that can beresolved through another data structure to obtain the full details, etc.For example, a keyvector containing Canny edge data for frame 1357,produced with a 5×5 blurring kernel may be named“KV_Edge_Canny_(—)1357_(—)5×5.”

To alert other processes of data that is in-process, a null entry can bewritten to the shared data structure when a function isinitialized—named in accordance with the function's final results. Thus,if the system starts to perform a Canny operation on frame 1357, with a5×5 blurring kernel, a null file may be written to the shared datastructure with the name noted above. (This can be performed by thefunction, or by the state machine—e.g., the dispatch process.) Ifanother process needs that information, and finds theappropriately-named file with a null entry, it knows such a process hasbeen launched. It can then monitor, or check back with, the shared datastructure and obtain the needed information when it becomes available.

More particularly, a process stage that needs that information wouldinclude among its input parameters a specification of a desired edgeimage—including descriptors giving its required qualities. The system(e.g., the dispatch process) would examine the types of data currentlyin memory (e.g., on the blackboard), and description tables, as noted,to determine whether appropriate data is presently available or inprocess. The possible actions could then include starting the stage withacceptable, available data; delay starting until a later time, when thedata is expected to be available; delay starting and schedule startingof a process that would generate the required data (e.g., Canny); ordelay or terminate the stage, due to lack of needed data and of theresources that would be required to generate them.

In considering whether alternate data is appropriate for use with aparticular operation, consideration may be given to data from otherframes. If the camera is in a free-running mode, it may be capturingmany (e.g., 30) frames every second. While an analysis process mayparticularly consider frame 1357 (in the example given above), it may beable to utilize information derived from frame 1356, or even frame 1200or 1500.

In this regard it is helpful to identify groups of frames encompassingimagery that is comparable in content. Whether two image frames arecomparable will naturally depend on the particular circumstances, e.g.,image content and operation(s) being performed.

In one exemplary arrangement, frame A may be regarded as comparable withframe B, if (1) a relevant region of interest appears in both frames(e.g., the same face subject, or barcode subject), and (2) if each ofthe frames between A and B also includes that same region of interest(this provides some measure of protection against the subject changingbetween when the camera originally viewed the subject, and when itreturned to the subject).

In another arrangement, two frames are deemed comparable if their colorhistograms are similar, to within a specified threshold (e.g., they havea correlation greater than 0.95, or 0.98).

In yet another arrangement, MPEG-like techniques can be applied to animage stream to determine difference information between two frames. Ifthe difference exceeds a threshold, the two frames are deemednon-comparable.

A further test, which can be imposed in addition to those criteria notedabove, is that a feature- or region-of-interest in the frame isrelatively fixed in position (“relatively” allowing a threshold ofpermitted movement, e.g., 10 pixels, 10% of the frame width, etc.).

A great variety of other techniques can alternatively be used; these arejust illustrative.

In one particular embodiment, the mobile device maintains a datastructure that identifies comparable image frames. This can be as simpleas a table identifying the beginning and ending frame of each group,e.g.:

Start Frame End Frame . . . . . . 1200 1500 1501 1535 1536 1664 . . . .. .

In some arrangements, a third field may be provided—indicating frameswithin the indicated range that are not, for some reason, comparable(e.g., out of focus).

Returning to the earlier-noted example, if a function desires input data“KV_Edge_Canny_(—)1357_(—)5×5” and none is found, it can expand thesearch to look for “KV_Edge_Canny_(—)1200_(—)5×5” through“KV_Edge_Canny_(—)1500_(—)5×5,” based on the comparability (roughequivalence) indicated by the foregoing table. And, as indicated, it mayalso be able to utilize edge data produced by other methods, again, fromany of frames 1200-1500.

Thus, for example, a barcode may be located by finding a region of highhorizontal contrast in frame 1250, and a region of low vertical contrastin frame 1300. After location, this barcode may be decoded by referenceto bounding line structures (edges) found in frame 1350, and correlationof symbol patterns found in frames 1360, 1362 and 1364. Because allthese frames are within a common group, the device regards data derivedfrom each of them to be usable with data derived from each of theothers.

In more sophisticated embodiments, feature tracking (flow) betweenframes can be discerned, and used to identify motion between frames.Thus, for example, the device can understand that a line beginning atpixel (100,100) in frame A corresponds to the same line beginning atpixel (101, 107) in frame B. (Again, MPEG techniques can be used, e.g.,for frame-to-frame object tracking.) Appropriate adjustments can be madeto re-register the data, or the adjustment can be introduced otherwise.

In simpler embodiments, equivalence between image frames is based simplyon temporal proximity. Frames within a given time-span (or frame-span)of the subject frame are regarded to be comparable. So in looking forCanny edge information for frame 1357, the system may accept edgeinformation from any of frames 1352-1362 (i.e., plus and minus fiveframes) to be equivalent. While this approach will sometimes lead tofailure, its simplicity may make it desirable in certain circumstances.

Sometimes an operation using substituted input data fails (e.g., itfails to find a barcode, or recognize a face) because the input datafrom the alternate process wasn't of the precise character of theoperation's nominal, desired input data. For example, although rare, aHough transform-based feature recognition might fail because the inputdata was not produced by the Canny operator, but by an alternateprocess. In the event an operation fails, it may be re-attempted—thistime with a different source of input data. For example, the Cannyoperator may be utilized, instead of the alternate. However, due to thecosts of repeating the operation, and the generally low expectation ofsuccess on the second try, such re-attempts are generally not undertakenroutinely. One case in which a re-attempt may be tried is if theoperation was initiated in top-down fashion, such as in response to useraction.)

In some arrangements, the initial composition of services decisionsdepend, in some measure, on whether an operation was initiated top-downor bottom-up (these concepts are discussed below). In the bottom-upcase, for example, more latitude may be allowed to substitute differentsources of input data (e.g., sources with less indicated correspondenceto the nominal data source) than in the top-down case.

Other factors that can be considered in deciding composition of servicemay include power and computational constraints, financial costs forcertain cloud-based operations, auction outcomes, user satisfactionrankings, etc.

Again, tables giving relative information for each of alternateoperations may be consulted to help the composition of servicesdecision. One example is shown in FIG. 6.

The FIG. 6 table gives metrics for CPU and memory required to executedifferent edge finding functions. The metrics may be actual values ofsome sort (e.g., CPU cycles to perform the stated operation on an imageof a given size, e.g., 1024×1024, and KB of RAM needed to execute suchan operation), or they may be arbitrarily scaled, e.g., on a scale of0-100.

If a function requires edge data—preferably from a Canny operation, andno suitable data is already available, the state machine must decidewhether to invoke the requested Canny operation, or another. If systemmemory is in scarce supply, the table of FIG. 6 (in conjunction with thetable of FIG. 5) suggests that an Infinite Symmetric Exponential filtermay be used instead: it is only slightly greater in CPU burden, buttakes 25% less memory. (FIG. 5 indicates the Infinite SymmetricExponential filter has a correspondence of 95 with Canny, so it shouldbe functionally substitutable.) Sobel and Kirch require much smallermemory footprints, but FIG. 5 indicates that these may not be suitable(scores of 70).

The real time state machine can consider a variety of parameters—such asthe scores of FIGS. 5 and 6, plus other scores for costs, usersatisfaction, current system constraints (e.g., CPU and memoryutilization), and other criteria, for each of the alternative edgefinding operations. These may be input to a process that weights andsums different combinations of the parameters in accordance with apolynomial equation. The output of this process yields a score for eachof the different operations that might be invoked. The operation withthe highest score (or the lowest, depending on the equation) is deemedthe best in the present circumstances, and is then launched by thesystem.

While the tables of FIGS. 5 and 6 considered just local device executionof such functions, cloud-based execution may also be considered. In thiscase, the processor and memory costs of the function are essentiallynil, but other costs may be incurred, e.g., in increased time to receiveresults, in consumption of network bandwidth, and possibly in financialmicropayment. Each of these costs may be different for alternativeservice providers and functions. To assess these factors, additionalscores can be computed, e.g., for each service provider and alternatefunction. These scores can include, as inputs, an indication of urgencyto get results back, and the increased turnaround time expected from thecloud-based function; the current usage of network bandwidth, and theadditional bandwidth that would be consumed by delegation of thefunction to a cloud-based service; the substitutability of thecontemplated function (e.g., Infinite Symmetric Exponential filter)versus the function nominally desired (e.g., Canny); and an indicationof the user's sensitivity to price, and what charge (if any) would beassessed for remote execution of the function. A variety of otherfactors can also be involved, including user preferences, auctionresults, etc. The scores resulting from such calculations can be used toidentify a preferred option among the different remoteproviders/functions considered. The system can then compare the winningscore from this exercise with the winning score from those associatedwith performance of a function by the local device. (Desirably, thescoring scales are comparable.) Action can then be taken based on suchassessment.

The selection of services can be based other factors as well. Fromcontext, indications of user intention, etc., a set of recognitionagents relevant to the present circumstances can be identified. Fromthese recognition agents the system can identify a set consisting oftheir desired inputs. These inputs may involve other processes whichhave other, different, inputs. After identifying all the relevantinputs, the system can define a solution tree that includes theindicated inputs, as well as alternatives. The system then identifiesdifferent paths through the tree, and selects one that is deemed (e.g.,based on relevant constraints) to be optimal. Again, both local andcloud-based processing can be considered.

One measure of optimality is a cost metric computed by assigningparameters to the probability that a solution will be found, and to theresources involved. The metric is then the quotient:

Cost=(Resources Consumed)/(Probability of Solution Being Found)

The state machine can manage compositing of RA services by optimizing(minimizing) this function. In so doing, it may work with cloud systemsto manage resources and calculate the costs of various solution treetraversals.

To facilitate this, RAs may be architected with multiple stages, eachprogressing towards the solution. They desirably should be granular intheir entry points and verbose in their outputs (e.g., exposing loggingand other information, indications re confidence of convergence, state,etc.). Often, RAs that are designed to use streaming data models arepreferred.

In such respects, the technology can draw from “planning models” knownin the field of artificial intelligence (AI), e.g., in connection with“smart environments.”

(The following discussion of planning models draws, in part, fromMarquardt, “Evaluating AI Planning for Service Composition in SmartEnvironments,” ACM Conf. on Mobile and Ubiquitous Media 2008, pp.48-55.)

A smart environment, as conceived by Mark Weiser at Xerox PARC, is onethat is “richly and invisibly interwoven with sensors, actuators,displays, and computational elements, embedded seamlessly in theeveryday objects of our lives, and connected through a continuousnetwork.” Such environments are characterized by dynamic ensembles ofdevices that offer individualized services (e.g., lighting, heating,cooling, humidifying, image projecting, alerting, image recording, etc.)to the user in an unobtrusive manner.

FIG. 7 is illustrative. The intentions of a user are identified, e.g.,by observation, and by reference to context. From this information, thesystem derives the user's presumed goals. The step of strategy synthesisattempts to find a sequence of actions that meets these goals. Finally,these actions are executed using the devices available in theenvironment.

Because the environment is changeable, the strategy synthesis—whichattends to composition of services—must be adaptable, e.g., as goals andavailable devices change. The composition of services task is regardedas an AI “planning” problem.

AI planning concerns the problem of identifying action sequences that anautonomous agent must execute in order to achieve a particular goal.Each function (service) that an agent can perform is represented as anoperator. (Pre- and post-conditions can be associated with theseoperators. Pre-conditions describe prerequisites that must be present toexecute the operator (function). Post-conditions describe the changes inthe environment triggered by execution of the operator—a change to whichthe smart environment may need to be responsive.) In planning terms, the“strategy synthesis” of FIG. 7 corresponds to plan generation, and the“actions” correspond to plan execution. The plan generation involvesservice composition for the smart environment.

A large number of planners is known from the AI field. See, e.g., Howe,“A Critical Assessment of Benchmark Comparison in Planning,” Journal ofArtificial Intelligence Research, 17:1-33, 2002. Indeed, there is anannual conference devoted to competitions between AI planners (seeipc<dot>icaps-conference<dot>org). A few planners for composing servicesin smart environments have been evaluated, in Amigoni, “What Planner forAmbient Intelligence Applications?” IEEE Systems, Man and Cybernetics,35(1):7-21, 2005. Other planners for service composition in smartenvironments are particularly considered in the Marquardt paper notedearlier, including UCPOP, SGP, and Blackbox. All generally use a variantof PDDL (Planning Domain Definition Language)—a popular descriptionlanguage for planning domains and problems.

Marquardt evaluated different planners in a simple smart environmentsimulation—a portion of which is represented by FIG. 8, employingbetween five and twenty devices—each with two randomly selectedservices, and randomly selected goals. Data are exchanged between themodel components in the form of messages along the indicated lines. Theservices in the simulation each have up to 12 pre-conditions (e.g.,“light_on,” “have_document_A,” etc.). Each service also has variouspost-conditions.

The study concluded that all three planners are satisfactory, but thatBlackbox (Kautz, “Blackbox: A New Approach to the Application of TheoremProving to Problem Solving,” AIPS 1998) performed best. Marquardt notedthat where the goal is not solvable, the planners generally took anundue amount of time trying unsuccessfully to devise a plan to meet thegoal. The authors concluded that it is better to terminate a planningprocess (or initiate a different planner) if the process does not yielda solution within one second, in order to avoid wasting resources.

Although from a different field of endeavor, applicants believe thislatter insight should likewise be applied when attempting composition ofservices to achieve a particular goal in the field of visual query: if asatisfactory path through a solution tree (or other planning procedure)cannot be devised quickly, the state machine should probably regard thefunction as insoluble with available data, and not expend more resourcestrying to find a solution. A threshold interval may be established insoftware (e.g., 0.1 seconds, 0.5 seconds, etc.), and a timer can becompared against this threshold and interrupt attempts at a solution ifno suitable strategy is found before the threshold is reached.

Embodiments of the present technology can also draw from work in thefield of web services, which increasingly are being included asfunctional components of complex web sites. For example, a travel website may use one web service to make an airline reservation, another toselect a seat on the airplane, and another to charge a user's creditcard. The travel web site needn't author these functional components; ituses a mesh of web services authored and provided by others. Thismodular approach—drawing on work earlier done by others—speeds systemdesign and delivery.

This particular form of system design goes by various names, includingService Oriented Architecture (SOA) and Service Oriented Computing.Although this style of design saves the developer from writing softwareto perform the individual component operations, there is still the taskof deciding which web services to use, and orchestrating the submissionof data to—and collection of results from—such services. A variety ofapproaches to these issues are known. See, e.g., Papazoglou,“Service-Oriented Computing Research Roadmap,” Dagstuhl SeminarProceedings 05462, 2006; and Bichler, “Service Oriented Computing,” IEEEComputer, 39:3, March, 2006, pp. 88-90.

Service providers naturally have a finite capacity for providingservices, and must sometimes deal with the problem of triaging requeststhat exceed their capacity. Work in this field includes algorithms forchoosing among the competing requests, and adapting charges for servicesin accordance with demand. See, e.g., Esmaeilsabzali et al, “OnlinePricing for Web Service Providers,” ACM Proc. of the 2006 Int'l Workshopon Economics Driven Software Engineering Research.

The state machine of the present technology can employ Service OrientedComputing arrangements to expand the functionality of mobile devices(for visual search and otherwise) by deploying part of the processingburden to remote servers and agents. Relevant web services may beregistered with one or more cloud-based broker processes, e.g.,specifying their services, inputs, and outputs in a standardized, e.g.,XML, form. The state machine can consult with such broker(s) inidentifying services to fulfill the system's needs. (The state machinecan consult with a broker of brokers, to identify brokers dealing withparticular types of services. For example, cloud-based service providersassociated with a first class of services, e.g., facial recognition, maybe cataloged by a first broker, while cloud-based service providersassociated with a different class of services, e.g., OCR, may becataloged by a second broker.)

The Universal Description Discovery and Integration (UDDI) specificationdefines one way for web services to publish, and for the state machineto discover, information about web services. Other suitable standardsinclude Electronic Business using eXtensible Markup Language (ebXML) andthose based on the ISO/IEC 11179 Metadata Registry (MDR). Semantic-basedstandards, such as WSDL-S and OWL-S (noted below), allow the statemachine to describe desired services using terms from a semantic model.Reasoning techniques, such as description logic inferences, can then beused to find semantic similarities between the description offered bythe state machine, and service capabilities of different web services,allowing the state machine to automatically select a suitable webservice. (As noted elsewhere, reverse auction models can be used, e.g.,to select from among several suitable web services.)

Intuitive Computing Platform (ICP) State Machine—Concurrent Processes

To maintain the system in a responsive state, the ICP state machine mayoversee various levels of concurrent processing (analogous tocognition), conceptually illustrated in FIG. 9. Four such levels, and arough abridgement of their respective scopes, are:

-   -   Reflexive—no user or cloud interaction    -   Conditioned—based on intent; minimal user interaction; engaging        cloud    -   Intuited, or “Shallow solution”—based on solutions arrived at on        device, aided by user interaction and informed by interpretation        of intent and history    -   “Deep Solution”—full solution arrived at through session with        user and cloud.

FIG. 10 further details these four levels of processing associated withperforming visual queries, organized by different aspects of the system,and identifying elements associated with each.

Reflexive processes typically take just a fraction of a second toperform. Some may be refreshed rarely (e.g., what is the cameraresolution). Others—such as assessing camera focus—may recur severaltimes a second (e.g., once or twice, up through tens of times—such asevery frame capture). The communications component may simply check forthe presence of a network connection. Proto-baubles (analog baubles) maybe placed based on gross assessments of image segmentation (e.g., isthere a bright spot?). Temporal aspects of basic image segmentation maybe noticed, such as flow—from one frame to the next, e.g., of a red blob3 pixels to the right. The captured 2D image is presented on the screen.The user typically is not involved at this level except, e.g., that userinputs—like tapped baubles—are acknowledged.

Conditioned processes take longer to perform (although typically lessthan a second), and may be refreshed, e.g., on the order of every halfsecond. Many of these processes relate to context data and acting onuser input. These include recalling what actions the user undertook thelast time in similar contextual circumstances (e.g., the user often goesinto Starbucks on the walk to work), responding to user instructionsabout desired verbosity, configuring operation based on the currentdevice state (e.g., airplane mode, power save mode), performingelementary orientation operations, determining geolocation, etc.Recognition agents that appear relevant to the current imagery and othercontext are activated, or prepared for activation (e.g., the image looksa bit like text, so prepare processes for possible OCR recognition).Recognition agents can take note of other agents that are also running,and can post results to the blackboard for their use. Baubles indicatingoutputs from certain operations appear on the screen. Hand-shaking withcloud-based resources is performed, to ready data channels for use, andquality of the channels is checked. For processes involving cloud-basedauctions, such auctions may be announced, together with relevantbackground information (e.g., about the user) so that differentcloud-based agents can decide whether to participate, and make anyneeded preparations.

Intuited processes take still longer to perform, albeit mostly on thedevice itself. These processes generally involve supporting therecognition agents in their work—composing needed keyvectors, presentingassociated UIs, invoking related functions, responding to and balancingcompeting requests for resources, etc. The system discerns what semanticinformation is desired, or may likely be desired, by the user. (If theuser, in Starbucks, typically images the front page of the New YorkTimes, then operations associated with OCR may be initiated—without userrequest. Likewise, if presentation of text-like imagery has historicallyprompted the user to request OCR′ing and translation into Spanish, theseoperations can be initiated—including readying a cloud-based translationengine.) Relevant ontologies may be identified and employed. Outputbaubles posted by recognition agents can be geometrically remapped inaccordance with the device's understanding of the captured scene, andother aspects of 3D understanding can be applied. A rules engine canmonitor traffic on the external data channels, and respond accordingly.Quick cloud-based responses may be returned and presented to theuser—often with menus, windows, and other interactive graphicalcontrols. Third party libraries of functions may also be involved atthis level.

The final Deep Solutions are open-ended in timing—they may extend fromseconds, to minutes, or longer, and typically involve the cloud and/orthe user. Whereas Intuited processes typically involve individualrecognition agents, Deep Solutions may be based on outputs from severalsuch agents, interacting, e.g., by association. Social network input mayalso be involved in the process, e.g., using information about peergroups, tastemakers the user respects, their histories, etc. Out in thecloud, elaborate processes may be unfolding, e.g., as remote agentscompete to provide service to the device. Some data earlier submitted tothe cloud may prompt requests for more, or better, data Recognitionagents that earlier suffered for lack of resources may now be allowedall the resources they want because other circumstances have made clearthe need for their output. A coveted 10×20 pixel patch adjacent to theStatue of Liberty is awarded to a happy bauble provider, who hasarranged a pleasing interactive experience to the user who taps there.Regular flows of data to the cloud may be established, to provideon-going cloud-based satisfaction of user desires. Other processes—manyinteractive—may be launched in this phase of operation as a consequenceof the visual search, e.g., establishing a Skype session, viewing aYouTube demonstration video, translating an OCR'd French menu intoEnglish, etc.

At device startup (or at other phases of its operation), the device maydisplay baubles corresponding to some or all of the recognition agentsthat it has available and ready to apply. This is akin to all thewarning lights illuminating on the dashboard of a car when firststarted, demonstrating the capability of the warning lights to work ifneeded (or akin to a player's display of collected treasure and weaponsin a multi-player online game—tools and resources from which the usermay draw in fighting dragons, etc.).

It will be recognized that this arrangement is illustrative only. Inother implementations, other arrangements can naturally be used.

Top-Down and Bottom-Up; Lazy Activation Structure

Applications may be initiated in various ways. One is by userinstruction (“top-down”).

Most applications require a certain set of input data (e.g.,keyvectors), and produce a set of output data (e.g., keyvectors). If auser instructs the system to launch an application (e.g., by tapping abauble, interacting with a menu, gesturing, or what not), the system canstart by identifying what inputs are required, such as by building a“keyvectors needed” list, or tree. If all the needed keyvectors arepresent (e.g., on the blackboard, or in a “keyvectors present” list ortree), then the application can execute (perhaps presenting a brightbauble) and generate the corresponding output data.

If all of the needed keyvectors are not present, a bauble correspondingto the application may be displayed, but only dimly. A reverse directoryof keyvector outputs can be consulted to identify other applicationsthat may be run in order to provide the keyvectors needed as input forthe user-initiated application. All of the keyvectors required by thoseother applications can be added to “keyvectors needed.” The processcontinues until all the keyvectors required by these other applicationsare in “keyvectors present.” These other applications are then run. Allof their resulting output keyvectors are entered into the “keyvectorspresent” list. Each time another keyvector needed for the top-levelapplication becomes available, the application's bauble may bebrightened. Eventually, all the necessary input data is available, andthe application initiated by the user is run (and a bright bauble mayannounce that fact).

Another way an application can be run is “bottom up”—triggered by theavailability of its input data. Rather than a user invoking anapplication, and then waiting for necessary data, the process isreversed. The availability of data drives the activation (and often thenselection) of applications. Related work is known under the “lazyevaluation” or “lazy activation” moniker.

One particular implementation of a lazy activation structure draws fromthe field of artificial intelligence, namely production systemarchitectures. Productions typically have two parts—a condition (IF),and an action (THEN). These may take the form of stored rules (e.g., ifan oval is present, then check whether a majority of the pixels insidethe oval have a skintone color). The condition may have severalelements, in logical combination (e.g., if an oval is present, and ifthe oval's height is at least 50 pixels, then . . . ); however, suchrules can often be broken down into a series of simpler rules, which maysometimes be preferable (e.g., if an oval is detected, then checkwhether the oval's height is at least 50 pixels; if the oval's height isat least 50 pixels, then . . . ).

The rules are evaluated against a working memory—a store that representsthe current state of the solution process (e.g., the blackboard datastructure).

When a rule stating a condition is met (matched), the action isgenerally executed—sometimes subject to deliberation. For example, ifseveral conditions are met, the system must further deliberate to decidein what order to execute the actions. (Executing one action—in somecases—may change other match conditions, so that different outcomes mayensue depending on how the deliberation is decided. Approaches todeliberation include, e.g., executing matched rules based on the orderthe rules are listed in a rule database, or by reference to differentpriorities assigned to different rules.)

These arrangements are sometimes termed match/deliberate (orevaluate)/execute arrangements (c.f., Craig, Formal Specifications ofAdvanced AI Architectures, Ellis Horward, Ltd., 1991). In some cases,the “match” step may be met by a user pressing a button, or by thesystem being in the bottom-up modality, or some other condition notexpressly tied to sensed content.

As noted, a conditional rule starts the process—a criterion that must beevaluated. In the present circumstances, the conditional rule may relateto the availability of a certain input data. For example, the “bottomup” process can be activated on a regular basis by comparing the current“keyvectors present” tree with the full list of top-level applicationsinstalled on the system. If any of an application's input requirementsare already present, it can launch into execution.

If some (but not all) of an application's input requirements are alreadypresent, a corresponding bauble may be displayed, in an appropriatedisplay region, at a brightness indicating how nearly all its inputs aresatisfied. The application may launch without user input once all itsinputs are satisfied. However, many applications may have a “useractivation” input. If the bauble is tapped by the user (or if another UIdevice receives a user action), the application is switched into thetop-down launch mode—initiating other applications—as described above—togather the remaining predicate input data, so that top level applicationcan then run.

In similar fashion, an application for which some (not all) inputs areavailable, may be tipped into top-down activation by circumstances, suchas context. For example, a user's historical pattern of activating afeature in certain conditions can serve as inferred user intent,signaling that the feature should be activated when those conditionsrecur. Such activation may occur even with no requisite inputsavailable, if the inferred user intent is compelling enough.

(In some implementations, traditional production system techniques maybe cumbersome due to the large number of rules being evaluated.Optimizations, such as a generalized trie pattern-matching approach fordetermining which rules' conditions are met, can be employed. See, e.g.,Forgy, “Rete: A Fast Algorithm for the Many Pattern/Many Object PatternMatch Problem,” Artificial Intelligence, Vol. 19, pp 17-37, 1982.)

In arrangements like the foregoing, resources are only applied tofunctions that are ready to run—or nearly so. Functions are launchedinto action opportunistically—when merited by the availability ofappropriate input data

Regularly-Performed Image Processing

Some user-desired operations will always be too complex to be performedby the portable system, alone; cloud resources must be involved.Conversely, there are some image-related operations that the portablesystem should be able to perform without any use of cloud resources.

To enable the latter, and facilitate the former, the system designer mayspecify a set of baseline image processing operations that are routinelyperformed on captured imagery, without being requested by a function orby a user. Such regularly-performed background functions may providefodder (output data, expressed as keyvectors) that other applicationscan use as input. Some of these background functions can also serveanother purpose: standardization/distillation of image-relatedinformation for efficient transfer to, and utilization by, other devicesand cloud resources.

A first class of such regularly-performed operations generally takes oneor more image frames (or parts thereof) as input, and produces an imageframe (or partial frame) keyvector as output. Exemplary operationsinclude:

-   -   Image-wide (or region of interest-wide) sampling or        interpolation: the output image may not have the same dimensions        as the source, nor is the pixel depth necessarily the same    -   Pixel remapping: the output image has the same dimensions as the        source, though the pixel depth need not be the same. Each source        pixel is mapped independently        -   examples: thresholding, ‘false color’, replacing pixel            values by exemplar values    -   Local operations: the output image has the same dimensions as        the source, or is augmented in a standard way (e.g., adding a        black image border). Each destination pixel is defined by a        fixed-size local neighborhood around the corresponding source        pixel        -   examples: 6×6 Sobel vertical edge, 5×5 line-edge magnitude,            3×3 local max, etc.    -   Spatial remapping: e.g., correcting perspective or curvature        ‘distortion’    -   EFT or other mapping into an “image” in a new space    -   Image arithmetic: output image is the sum, maximum, etc of input        images        -   Sequence averaging: each output image averages k-successive            input images        -   Sequence (op)ing: each output image is a function of            k-successive input images

A second class of such background operations processes one or more inputimages (or parts thereof) to yield an output keyvector consisting of alist of 1D or 2D regions or structures. Exemplary operations in thissecond class include:

-   -   Long-line extraction: returns a list of extracted straight line        segments (e.g., expressed in a slope-intercept format, with an        endpoint and length)    -   A list of points where long lines intersect (e.g., expressed in        row/column format)    -   Oval finder: returns a list of extracted ovals (in this, and        other cases, location and parameters of the noted features are        included in the listing)    -   Cylinder finder: returns a list of possible 3D cylinders (uses        Long-line)    -   Histogram-based blob extraction: returns a list of image regions        which are distinguished by their local histograms    -   Boundary-based blob extraction: returns a list of image regions        which are distinguished by their boundary characteristics    -   Blob ‘tree’ in which each component blob (including the full        image) has disjoint sub-blobs which are fully contained in it.        Can carry useful scale-invariant (or at least scale-resistant)        information        -   example: the result of thresholding an image at multiple            thresholds    -   Exact boundaries, e.g., those of thresholded blob regions    -   Indistinct boundaries, e.g., a list of edges or points which        provide a reasonably dense region boundary, but may have small        gaps or inconsistencies, unlike the boundaries of thresholded        blobs

A third class of such routine, on-going processes produces a table orhistogram as output keyvector data Exemplary operations in this thirdclass include:

-   -   Histogram of hue, intensity, color, brightness, edge value,        texture, etc.    -   2D histogram or table indicating feature co-occurrence, e.g., of        1D values: (hue, intensity), (x-intensity, y-intensity), or some        other pairing

A fourth class of such default image processing operations consists ofoperations on common non-image objects. Exemplary operations in thisfourth class include:

-   -   Split/merge: input blob list yields a new, different blob list    -   Boundary repair: input blob list yields a list of blobs with        smoother boundaries    -   Blob tracking: a sequence of input blob lists yields a list of        blob sequences    -   Normalization: image histogram and list of histogram-based blobs        returns a table for remapping the image (perhaps to “region        type” values and “background” value(s))

The foregoing operations, naturally, are only exemplary. There are many,many other low-level operations that can be routinely performed. Afairly large set of the types above, however, are generally useful,demand a reasonably small library, and can be implemented withincommonly-available CPU/GPU requirements.

Contextually-Triggered Image Processing; Barcode Decoding

The preceding discussion noted various operations that the system mayperform routinely, to provide keyvector data that can serve as input fora variety of more specialized functions. Those more specializedfunctions can be initiated in a top-down manner (e.g., by userinstruction), or in bottom-up fashion (e.g., by the availability of alldata predicates).

In addition to the operations just-detailed, the system may also launchprocesses to generate other keyvectors based on context.

To illustrate, consider location. By reference to geolocation data, adevice may determine that a user is in a grocery store. In this case thesystem may automatically start performing additional image processingoperations that generate keyvector data which may be useful forapplications commonly relevant in grocery stores. (These automaticallytriggered applications may, in turn, invoke other applications that areneeded to provide inputs for the triggered applications.)

For example, in a grocery store the user may be expected to encounterbarcodes. Barcode decoding includes two different aspects. The first isto find a barcode region within the field of view. The second is todecode the line symbology in the identified region. Operationsassociated with the former aspect can be undertaken routinely when theuser is determined to be in a grocery store (or other retailestablishment). That is, the routinely-performed set of image processingoperations earlier detailed is temporarily enlarged by addition of afurther set of contextually-triggered operations—triggered by the user'slocation in the grocery store.

Finding a barcode can be done by analyzing a greyscale version ofimagery to identify a region with high image contrast in the horizontaldirection, and low image contrast in the vertical direction. Thus, whenin a grocery store, the system may enlarge the catalog of imageprocessing operations that are routinely performed, to also includecomputation of a measure of localized horizontal greyscale imagecontrast, e.g., 2-8 pixels to either side of a subject pixel. (One suchmeasure is summing the absolute values of differences in values ofadjacent pixels.) This frame of contrast information (or a downsampledframe) can comprise a keyvector—labeled as to its content, and postedfor other processes to see and use. Similarly, the system can computelocalized vertical grayscale image contrast, and post those results asanother keyvector.

The system may further process these two keyvectors by, for each pointin the image, subtracting the computed measure of local vertical imagecontrast from the computed measure of local horizontal image contrast.Normally, this operation yields a chaotic frame of data—at pointsstrongly positive, and at points strongly negative. However, in barcoderegions it is much less chaotic—having a strongly positive value acrossthe barcode region. This data, too, can be posted for other processes tosee, as yet another (third) keyvector that is routinely produced whilethe user is in the grocery store.

A fourth keyvector may be produced from the third, by applying athresholding operation—identifying only those points having a value overa target value. This operation thus identifies the points in the imagethat seem potentially barcode-like in character, i.e., strong inhorizontal contrast and weak in vertical contrast.

A fifth keyvector may be produced from the fourth, by applying aconnected component analysis—defining regions (blobs) of points thatseem potentially barcode-like in character.

A sixth keyvector may be produced by the fifth—consisting of threevalues: the number of points in the largest blob; and the locations ofthe upper left and lower right corners of that blob (defined in row andcolumn offsets from the pixel at the upper left-most corner of the imageframe).

These six keyvectors are produced prospectively—without a user expresslyrequesting them, just because the user is in a location associated witha grocery store. In other contexts, these keyvectors would not normallybe produced.

These six operations may comprise a single recognition agent (i.e., abarcode locating agent). Or they may be part of a larger recognitionagent (e.g., a barcode locating/reading agent), or they may besub-functions that individually, or in combinations, are their ownrecognition agents.

(Fewer or further operations in the barcode reading process may besimilarly performed, but these six illustrate the point.)

A barcode reader application may be among those loaded on the device.When in the grocery store, it may hum along at a very low level ofoperation—doing nothing more than examining the first parameter in theabove-noted sixth keyvector for a value in excess of, e.g., 15,000. Ifthis test is met, the barcode reader may instruct the system to presenta dim barcode-indicating bauble at the location in the frame midwaybetween the blob corner point locations identified by the second andthird parameters of this sixth keyvector. This bauble tells the userthat the device has sensed something that might be a barcode, and thelocation in the frame where it appears.

If the user taps that dim bauble, this launches (top-down) otheroperations needed to decode a barcode. For example, the region of theimage between the two corner points identified in the sixth keyvector isextracted—forming a seventh keyvector.

A series of further operations then ensues. These can include filteringthe extracted region with a low frequency edge detector, and using aHough transform to search for nearly vertical lines.

Then, for each row in the filtered image, the position of the start,middle and end barcode patterns are identified through correlation, withthe estimated right and left edges of the barcode used as guides. Thenfor each barcode digit, the digit's position in the row is determined,and the pixels in that position of the row are correlated with possibledigit codes to determine the best match. This is repeated for eachbarcode digit, yielding a candidate barcode payload. Parity and checkdigit tests are then executed on the results from that row, and anoccurrence count for that payload is incremented. These operations arethen repeated for several more rows in the filtered image. The payloadwith the highest occurrence count is then deemed the correct barcodepayload.

At this point, the system can illuminate the barcode's baublebrightly—indicating that data has been satisfactorily extracted. If theuser taps the bright bauble, the device can present a menu of actions,or can launch a default action associated with a decoded barcode.

While in the arrangement just-described, the system stops its routineoperation after generating the sixth keyvector, it could have proceededfurther. However, due to resource constraints, it may not be practicalto proceed further at every opportunity, e.g., when the first parameterin the sixth keyvector exceeds 15,000.

In one alternative arrangement, the system may proceed further onceevery, e.g., three seconds. During each three second interval, thesystem monitors the first parameter of the sixth keyvector—looking for(1) a value over 15,000, and (2) a value that exceeds all previousvalues in that three second interval. When these conditions are met, thesystem can buffer the frame, perhaps overwriting any previously-bufferedframe. At the end of the three second interval, if a frame is buffered,it is the frame having the largest value of first parameter of any inthat three second interval. From that frame the system can then extractthe region of interest, apply the low frequency edge detector, findlines using a Hough procedure, etc., etc.—all the way through brightlyilluminating the bauble if a valid barcode payload is successfullydecoded.

Instead of rotely trying to complete a barcode reading operation everythree seconds, the system can do so opportunistically—when theintermediate results are especially promising.

For example, while the barcode reading process may proceed whenever thenumber of points in the region of interest exceeds 15,000, that value isa minimum threshold at which a barcode reading attempt might befruitful. The chance of reading a barcode successfully increases as thisregion of points becomes larger. So instead of proceeding furtherthrough the decoding process once every three seconds, furtherprocessing may be triggered by the occurrence of a value in excess of50,000 (or 100,000, or 500,000, etc.) in the first parameter of thesixth keyvector.

Such a large value indicates that an apparent barcode occupies asubstantial part of the camera's viewing frame. This suggests adeliberate action by the user—capturing a good view of a barcode. Inthis case, the remainder of the barcode reading operations can belaunched. This affords an intuitive feel to the device's behavior: theuser apparently intended to image a barcode, and the system—without anyother instruction—launched the further operations required to complete abarcode reading operation.

In like fashion, the system can infer—from the availability of imageinformation particularly suited to a certain type of operation—that theuser intends, or would benefit from, that certain type of operation. Itcan then undertake processing needed for that operation, yielding anintuitive response. (Text-like imagery can trigger operations associatedwith an OCR process; face-like features can trigger operationsassociated with facial recognition, etc.)

This can be done regardless of context. For example, a device canperiodically check for certain clues about the present environment,e.g., occasionally checking horizontal vs. vertical greyscale contrastin an image frame—in case barcodes might be in view. Although suchoperations may not be among those routinely loaded or loaded due tocontext, they can be undertaken, e.g., once every five seconds or soanyway, since the computational cost is small, and the discovery ofvisually useful information may be valued by the user.

Back to context, just as the system automatically undertook a differentset of background image processing operations because the user'slocation was in a grocery, the system can similarly adapt its set ofroutinely-occurring processing operations based on other circumstances,or context.

One is history (i.e., of the user, or of social peers of the user).Normally we may not use barcode readers in our homes. However, a bookcollector may catalog new books in a household library by reading theirISBN barcodes. The first time a user employs the device for thisfunctionality in the home, the operations generating the first-sixthkeyvectors noted above may need to be launched in top-downfashion—launched because the user indicates interest in reading barcodesthrough the device's UI. Likewise the second time. Desirably, however,the system notes the repeated co-occurrence of (1) the user at aparticular location, i.e., home, and (2) activation of barcode readingfunctionality. After such historical pattern has been established, thesystem may routinely enable generation of the first-sixth keyvectorsnoted above whenever the user is at the home location.

The system may further discern that the user activates barcode readingfunctionality at home only in the evenings. Thus, time can also beanother contextual factor triggering auto-launching of certain imageprocessing operations, i.e., these keyvectors are generated when theuser is at home, in the evening.

Social information can also provide triggering data. The user maycatalog books only as a solitary pursuit. When a spouse is in the house,the user may not catalog books. The presence of the spouse in the housemay be sensed in various manners. One is by Bluetooth radio signalsbroadcast from the spouse's cell phone. Thus, the barcode-locatingkeyvectors may be automatically generated when (1) the user is at home,(2) in the evenings, (3) without proximity to the user's spouse. If thespouse is present, or if it is daytime, or if the user is away from home(and the grocery), the system may not routinely generate the keyvectorsassociated with barcode-locating.

Bayesian or other statistical models of user behavior can be compiledand utilized to detect such co-occurrence of repeated circumstances, andthen be used to trigger actions based thereon.

(In this connection, the science of branch prediction in microprocessordesign can be informative. Contemporary processors include pipelinesthat may comprise dozens of stages—requiring logic that fetchesinstructions to be used 15 or 20 steps ahead. A wrong guess can requireflushing the pipeline—incurring a significant performance penalty.Microprocessors thus include branch prediction registers, which trackhow conditional branches were resolved, e.g., the last 255 times. Basedon such historical information, performance of processors is greatlyenhanced. In similar fashion, tracking historical patterns of deviceusage—both by the user and proxies (e.g., the user's social peers, ordemographic peers), and tailoring system behavior based on suchinformation, can provide important performance improvements.)

Audio clues (discussed further below) may also be involved in theauto-triggering of certain image processing operations. If auditoryclues suggest that the user is outdoors, one set of additionalbackground processing operations can be launched; if the clues suggestthe user is driving, a different set of operations can be launched.Likewise if the audio has hallmarks of a television soundtrack, or ifthe audio suggests the user is in an office environment. The softwarecomponents loaded and running in the system can thus adapt automaticallyin anticipation of stimuli that may be encountered—or operations theuser may request—in that particular environment. (Similarly, in ahearing device that applies different audio processing operations togenerate keyvectors needed by different audio functions, informationsensed from the visual environment can indicate a context that dictatesenablement of certain audio processing operations that may not normallybe run.)

Environmental clues can also cause certain functions to be selected,launched, or tailored. If the device senses the ambient temperature isnegative ten degrees Celsius, the user is presumably outdoors, inwinter. If facial recognition is indicated (e.g., by user instruction,or by other clue), any faces depicted in imagery may be bundled in hatsand/or scarves. A different set of facial recognition operations maythus be employed—taking into account the masking of certain parts of theface—than if, e.g., the context is a hot summer day, when people's hairand ears are expected to be exposed.

Other user interactions with the system can be noted, and lead toinitiation of certain image processing operations that are not normallyrun—even if the noted user interactions do not involve such operations.Consider a user who queries a web browser on the device (e.g., by textor spoken input) to identify nearby restaurants. The query doesn'tinvolve the camera or imagery. However, from such interaction, thesystem may infer that the user will soon (1) change location, and (2) bein a restaurant environment. Thus, it may launch image processingoperations that may be helpful in, e.g., (1) navigating to a newlocation, and (2) dealing with a restaurant menu.

Navigation may be aided by pattern-matching imagery from the camera withcurbside imagery along the user's expected route (e.g., from GoogleStreetview or other image repository, using SIFT). In addition toacquiring relevant imagery from Google, the device can initiate imageprocessing operations associated with scale-invariant feature transformoperations.

For example, the device can resample image frames captured by the cameraat different scale states, producing a keyvector for each. To each ofthese, a Difference of Gaussians function may be applied, yieldingfurther keyvectors. If processing constraints allow, these keyvectorscan be convolved with blur filters, producing still further keyvectors,etc.—all in anticipation of possible use of SIFT pattern matching.

In anticipation of viewing a restaurant menu, operations incident to OCRfunctionality can be launched.

For example, while the default set of background image processingoperations includes a detector for long edges, OCR requires identifyingshort edges. Thus, an algorithm that identifies short edges may belaunched; this output can be expressed in a keyvector.

Edges that define closed contours can be used to identifycharacter-candidate blobs. Lines of characters can be derived from thepositions of these blobs, and skew correction can be applied. From theskew-corrected lines of character blobs, candidate word regions can bediscerned. Pattern matching can then be applied to identify candidatetexts for those word regions. Etc., Etc.

As before, not all of these operations may be performed on everyprocessed image frame. Certain early operations may be routinelyperformed, and further operations can be undertaken based on (1) timingtriggers, (2) promising attributes of the data processed so far, (3)user direction, or (4) other criteria.

Back to the grocery store example, not only can context influence thetypes of image processing operations that are undertaken, but also themeaning to be attributed to different types of information (both imageinformation as well as other information, e.g., geolocation).

Consider a user's phone that captures a frame of imagery in a grocery.The phone may immediately respond—suggesting that the user is facingcans of soup. It can do this by referring to geolocation data andmagnetometer (compass) data, together with stored information about thelayout of that particular store—indicating the camera is facing shelvesof soups. A bauble, in its initial stages, may convey this first guessto the user, e.g., by an icon representing a grocery item, or by text,or by linked information.

An instant later, during initial processing of the pixels in thecaptured frame, the device may discern a blob of red pixels next to ablob of white pixels. By reference to a reference data source associatedwith the grocery store context (and, again, perhaps also relying on thegeolocation and compass data), the device may quickly guess (e.g., inless than a second) that the item is (most likely) a can of Campbell'ssoup, or (less likely) a bottle of ketchup. A rectangle may besuperimposed on the screen display—outlining the object(s) beingconsidered by the device.

A second later, the device may have completed an OCR operation on largecharacters on the white background, stating TOMATO SOUP—lending furthercredence to the Campbell's soup hypothesis. After a short furtherinterval, the phone may have managed to recognize the stylized script“Campbell's” in the red area of the imagery—confirming that the objectis not a store brand soup that is imitating the Campbell's color scheme.In a further second, the phone may have decoded a barcode visible on anearby can, detailing the size, lot number, manufacture date, and/orother information relating to the Campbell's Tomato Soup. At each stage,the bauble—or linked information—evolves in accordance with the device'srefined understanding of the object towards which the camera ispointing. (At any point the user can instruct the device to stop itsrecognition work—perhaps by a quick shake—preserving battery and otherresources for other tasks.)

In contrast, if the user is outdoors (sensed, e.g., by GPS, and/orbright sunshine), the phone's initial guess concerning a blob of redpixels next to a blob of white pixels will likely not be a Campbell'ssoup can. Rather, it may more likely guess it to be a U.S. flag, or aflower, or an article of clothing, or a gingham tablecloth—again byreference to a data store of information corresponding to the outdoorscontext.

Intuitive Computing Platform (ICP) Context Engine, Identifiers

Arthur C. Clarke is quoted as having said “Any sufficiently advancedtechnology is indistinguishable from magic.” “Advanced” can have manymeanings, but to imbue mobile devices with something akin to magic, thepresent specification interprets the term as “intuitive” or “smart.”

An important part of intuitive behavior is the ability to sense—and thenrespond to—the user's probable intent. As shown in FIG. 11, intent is afunction not only of the user, but also of the user's past.Additionally, intent can also be regarded as a function of activities ofthe user's peers, and their pasts.

In determining intent, context is a key. That is, context informs thededuction of intent, in the sense that knowing, e.g., where the user is,what activities the user and others have engaged in the last time atthis location, etc., is valuable in discerning the user's likelyactivities, needs and desires at the present moment. Such automatedreasoning about a user's behavior is a core goal of artificialintelligence, and much has been written on the subject. (See, e.g.,Choudhury et al, “Towards Activity Databases: Using Sensors andStatistical Models to Summarize People's Lives,” IEEE Data Eng. Bull,29(1): 49-58, March, 2006.)

Sensor data, such as imagery, audio, motion information, location, andBluetooth signals, are useful in inferring a user's likely activity (orin excluding improbable activities). As noted in Choudhury, such datacan be provided to a software module that processes the sensorinformation into features that can help discriminate between activities.Features can include high level information (such as identification ofobjects in the surroundings, or the number of people nearby, etc.), orlow level information (such as audio frequency content or amplitude,image shapes, correlation coefficients, etc.). From such features, acomputational model can deduce probable activity (e.g., walking,talking, getting coffee, etc.).

Desirably, sensor data from the phone is routinely logged, so patternsof historical activity can be discerned. In turn, activities that theuser undertakes can be noted, and correlated with the contexts (bothconcurrent and immediately preceding) that gave rise to such activities.Activities, in turn, are fodder from which user interests may beinferred. All such data is stored, and serves as a body of referenceinformation allowing the phone to deduce possible conduct in which theuser may engage in a given context, and discern which of the user'sinterests may be relevant in those circumstances.

Such intelligence may be codified in template, model or rule-base form(e.g., detailing recurring patterns of context data, and userconduct/interest apparently correlated with same—perhaps with associatedconfidence factors). Given real-time sensor data, such templates canprovide advice about expected intent to the portable device, so it canrespond accordingly.

These templates may be continuously refined—correlating with additionalaspects of context (e.g., season, weather, nearby friends, etc.) as moreexperience is logged, and more nuanced patterns can be discerned.Techniques familiar from expert systems may be applied in implementingthese aspects of the technology.

In addition to the wealth of data provided by mobile device sensors,other features useful in understanding context (and thus intent) can bederived from nearby objects. A tree suggests an outdoor context; atelevision suggests an indoor context. Some objects have associatedmetadata—greatly advancing contextual understanding. For example, someobjects within the user's environment may have RFIDs or the like. TheRFIDs convey unique object IDs. Associated with these unique object IDs,typically in a remote data store, are fixed metadata about the object towhich the RFIDs are attached (e.g., color, weight, ownership,provenance, etc). So rather than trying to deduce relevant informationfrom pixels alone, sensors in the mobile device—or in the environment,to which the mobile device links—can sense these carriers ofinformation, obtain related metadata, and use this information inunderstanding the present context.

(RFIDs are exemplary only; other arrangements can also be employed,e.g., digital watermarking, barcodes, fingerprinting, etc.)

Because user activities are complex, and neither object data nor sensordata lends itself to unambiguous conclusions, computational models forinferring the user's likely activity, and intent, are commonlyprobabilistic. Generative techniques can be used (e.g., Bayesian, hiddenMarkov, etc.). Discriminative techniques for class boundaries (e.g.,posterior probability) can also be employed. So too with relationalprobabilistic and Markov network models. In these approaches,probabilities can also depend on properties of others in the user'ssocial group(s).

In one particular arrangement, the determination of intent is based onlocal device observations relevant to context, mapped against templates(e.g., derived from the user's history, or from that of social friends,or other groups, etc.) that may be stored in the cloud.

By discerning intent, the present technology reduces the search-space ofpossible responses to stimuli, and can be used to segment input data todiscern activities, objects and produce identifiers. Identifiers can beconstructed with explicit and derived metadata.

To back up a bit, it is desirable for every content object to beidentified. Ideally, an object's identifier would be globally unique andpersistent. However, in mobile device visual query, this ideal is oftenunattainable (except in the case, e.g., of objects bearing machinereadable indicia, such as digital watermarks). Nonetheless, within avisual query session, it is desirable for each discerned object to havean identifier that is unique within the session.

One possible construct of a unique identifier (UID) includes two orthree (or more) components. One is a transaction ID, which may be asession ID. (One suitable session ID is a pseudo-random number, e.g.,produced by a PRN generator seeded with a device identifier, such as aMAC identifier. In other arrangements, the session ID can conveysemantic information, such as the UNIX time at which the sensor mostrecently was activated from an off, or sleep, state). Such a transactionID serves to reduce the scope needed for the other identificationcomponents, and helps make the identifier unique. It also places theobject identification within the context of a particular session, oraction.

Another component of the identifier can be an explicit object ID, whichmay be the clump ID referenced earlier. This is typically an assignedidentifier. (If a clump is determined to include several distinctlyidentifiable features or objects, further bits can be appended to theclump ID to distinguish same.)

Yet another component can be derived from the object, or circumstances,in some fashion. One simple example is a “fingerprint”—statisticallyunique identification information (e.g., SIFT, image signature, etc.)derived from features of the object itself. Additionally oralternatively, this component may consist of information relating tocontext, intent, deduced features—essentially anything that can be usedby a subsequent process to assist in the determination of identity. Thisthird component may be regarded as derived metadata, or “aura”associated with the object.

The object identifier can be a concatenation, or other combination, ofsuch components.

Pie Slices, Etc.

The different recognition processes invoked by the system can operate inparallel, or in cyclical serial fashion. In the latter case a clocksignal or the like may provide a cadence by which different of the pieslices are activated.

FIG. 12 shows such a cyclical processing arrangement as a circle of pieslices. Each slice represents a recognition agent process, or anotherprocess. The arrows indicate the progression from one to the next. Asshown by the expanded slice to the right, each slice can include severaldistinct stages, or states.

An issue confronted by the present technology is resource constraints.If there were no constraints, a seeing/hearing device could apply myriadresource-intensive recognition algorithms to each frame and sequence ofincoming data, constantly—checking each for every item of potentialinterest to the user.

In the real world, processing has costs. The problem can be phrased asone of dynamically identifying processes that should be applied to theincoming data, and dynamically deciding the type and quantity ofresources to devote to each.

In FIG. 12, different stages of the pie slice (recognition agentprocess) correspond to further levels of resource consumption. Theinnermost (pointed) stage generally uses the least resources. Thecumulative resource burden increases with processing by successivestages of the slice. (Although each stage will often be moreresource-intensive than those that preceded it, this is not required.)

One way this type of behavior can be achieved is by implementingrecognition and other operations as “cascaded sequences of operations,”rather than as monolithic operations. Such sequences frequently involveinitial operations with relatively low overheads, which—whensuccessful—can be continued by operations which may require moreresources, but are now only initiated after an initial indicator oflikely success. The technique can also facilitate opportunisticsubstitution of already available keyvectors for related featuresnormally used by an operation, again decreasing resource overhead asnoted earlier.

Consider, for discussion purposes, a facial recognition agent. Toidentify faces, a sequence of tests is applied. If any fails, then it isunlikely a face is present.

An initial test (common to many processes) is to check whether theimagery produced by the camera has features of any sort (vs., e.g., thecamera output when in a dark purse or pocket). This may be done by asimple histogram analysis of grey-scale pixel values for a sparsesampling of pixel locations across the image. If the histogram analysisshows all of the sampled pixels have substantially the same grey-scaleoutput, then further processing can be skipped.

If the histogram shows some diversity in pixel grey-scale values, thenthe image can next be checked for edges. An image without discernibleedges is likely an unusable image, e.g., one that is highly blurred orout-of-focus. A variety of edge detection filters are familiar to theartisan, as indicated above.

If edges are found, the facial detection procedure may next checkwhether any edge is curved and defines a closed region. (The ovalfinder, which runs as a routine background operation in certainimplementations, may allow the process to begin at this step.)

If so, a color histogram may be performed to determine whether asignificant percentage of pixels within the closed region are similar inhue to each other (skin comprises most of the face). “Significant” maymean greater than 30%, 50%, 70%, etc. “Similar” may mean within adistance threshold or angular rotation in a CIELAB sense. Tests forcolor within predefined skin tone ranges may optionally be applied.

Next, a thresholding operation may be applied to identify the darkest 5%of the pixels within the closed region. These pixels can be analyzed todetermine if they form groupings consistent with two eyes.

Such steps continue, in similar fashion, through the generation ofeigenvectors for the candidate face(s). (Facial eigenvectors arecomputed from the covariance matrix of the probability distribution ofthe high-dimensional vector space representation of the face.) If so,the eigenvectors may be searched for a match in a reference datastructure—either local or remote.

If any of the operations yields a negative result, the system canconclude that no discernible face is present, and terminate furtherface-finding efforts for that frame.

All of these steps can form stages in a single pie slice process.Alternatively, one or more steps may be regarded as elemental, anduseful to several different processes. In such case, such step(s) maynot form part of a special purpose pie slice process, but instead can beseparate. Such step(s) can be implemented in one or more pie sliceprocesses—cyclically executing with other agent processes and postingtheir results to the blackboard (whether other agents can find them). Orthey can be otherwise implemented.

In applying the system's limited resources to the different on-goingprocesses, detection state can be a useful concept. At each instant, thegoal sought by each agent (e.g., recognizing a face) may seem more orless likely to be reached. That is, each agent may have an instantaneousdetection state on a continuum, from very promising, through neutral,down to very discouraging. If the detection state is promising, moreresources may be allocated to the effort. If its detection state tendstowards discouraging, less resources can be allocated. (At some point, athreshold of discouragement may be reached that causes the system toterminate that agent's effort.) Detection state can be quantifiedperiodically by a software routine (separate, or included in the agentprocess) that is tailored to the particular parameters with which theagent process is concerned.

Some increased allocation of resources tends to occur when successivestages of agent processing are invoked (e.g., an FFT operation—whichmight occur in a 7^(th) stage, is inherently more complex than ahistogram operation—which might occur in a 4^(th) stage). But the systemcan also meter allocation of resources apart from base operationalcomplexity. For example, a given image processing operation might beperformed on either the system's CPU, or the GPU. An FFT might beexecuted with 1 MB of scratchpad memory for calculation, or 10 MB. Aprocess might be permitted to use (faster-responding) cache data storagein some circumstances, but only (slower-responding) system memory inothers. One stage may be granted access to a 4G network connection inone instance, but a slower 3G or WiFi network connection in another. Aprocess can publish information detailing these different options thatmay be invoked to increase its effectiveness, or to reduce its resourceconsumption (e.g., I can do X with this amount of resources; Y with thisfurther amount; Z with this lesser amount; etc.). Partial executionscenarios may be expressly offered. The state machine can select fromamong these options based on the various resource allocation factors.Processes that yield most promising results, or offer the possibility ofthe most promising results, can be granted privileged status inconsumption of system resources.

In a further arrangement, not only does allocation of resources dependon the agent's state in achieving its goal, but also its speed oracceleration to that end. For example, if promising results areappearing quickly in response to an initial resource effort level, thennot only can additional resources be applied, but more additionalresources can be applied than if the promising results appeared lessquickly. Allocation of resources can thus depend not only on detectionstate (or other metric of performance or result), but also on a first-or higher-order derivative of such a measure.

Relatedly, data produced by one stage of a detection agent process maybe so promising that the process can jump ahead one or morestages—skipping intervening stages. This may be the case, e.g., wherethe skipped stage(s) doesn't produce results essential to the process,but is undertaken simply to gain greater confidence that processing bystill further stages is merited. For example, a recognition agent mayperform stages 1, 2 and 3 and then—based a confidence metric from theoutput of stage 3—skip stage 4 and execute stage 5 (or skip stages 4 and5 and execute stage 6, etc.). Again, the state machine can exercise suchdecision-making control, based on a process' publication of informationabout different entry stages for that process.

The artisan will recognize that such an arrangement is different thanfamiliar prior art. Previously, different platforms offeredsubstantially different quanta of computing, e.g., mainframe, PC, cellphone, etc. Similarly, software was conceived as monolithic functionblocks, with fixed resource demands (E.g., a particular DLL may or maynot be loaded, depending on memory availability.) Designers thuspieced-together computing environments with blocks of established sizes.Some fit, others didn't. Foreign was the present concept of describingtasks in terms of different entry points and different costs, so that asystem could make intelligent decisions about how deep into a range offunctional capabilities it should go. Previously the paradigm was “Youmay run this function if you're able.” (Costs might be determinableafter the fact.) The present model shifts the paradigm to more like“I'll buy 31 cents of this function. Based on how things go, maybe I'llbuy more later.” In the present arrangement, a multi-dimensional rangeof choices is thus presented for performing certain tasks, from whichthe system can make intelligent decisions in view of other tasks,current resource constraints and other factors.

The presently described arrangement also allows the operating system toforesee how resource consumption will change with time. It may note, forexample, that promising results are quickly appearing in a particularrecognition agent, which will soon lead to an increased allocation ofresources to that agent. It may recognize that the apparently imminentsatisfactory completion of that agent's tasks will meet certain rules'conditions—triggering other recognition agents, etc. In view of theforthcoming spike in resource consumption the operating system maypro-actively take other steps, e.g., throttling back the wirelessnetwork from 4G to 3G, more aggressively curtailing processes that arenot yielding encouraging results, etc. Such degree of foresight andresponsiveness is far richer than that associated with typicalbranch-prediction approaches (e.g., based on rote examination of thelast 255 outcomes of a particular branch decision).

Just as resource allocation and stage-skipping can be prompted bydetection state, they can also be prompted by user input. If the userprovides encouragement for a particular process, that process can beallocated extra resources, and/or may continue beyond a point at whichits operation might otherwise have been automatically curtailed for lackof promising results. (E.g., if the detection state continuum earliernoted runs from scores of 0<wholly discouraging> to 100< whollyencouraging>, and the process normally terminates operation if its scoredrops below a threshold of 35, then that threshold may be dropped to 25,or 15, if the user provides encouragement for that process. The amountof threshold change can be related to an amount of encouragementreceived.)

The user encouragement can be express or implied. An example of expressencouragement is where the user provides input signals (e.g., screentaps, etc.), instructing that a particular operation be performed (e.g.,a UI command instructing the system to process an image to identify thedepicted person).

In some embodiments the camera is continuously capturingimages—monitoring the visual environment without particular userinstruction. In such case, if the user activates a shutter button or thelike, then that action can be interpreted as evidence of express userencouragement to process the imagery framed at that instant.

One example of implied encouragement is where the user taps on a persondepicted in an image. This may be intended as a signal to learn moreabout the person, or it may be a random act. Regardless, it issufficient to cause the system to increase resource allocation toprocesses relating to that part of the image, e.g., facial recognition.(Other processes may also be prioritized, e.g., identifying a handbag,or shoes, worn by the person, and researching facts about the personafter identification by facial recognition—such as through use of asocial network, e.g., LinkedIn or Facebook; through use of Google,pipl<dot>com, or other resource.)

The location of the tap can be used in deciding how much increase inresources should be applied to different tasks (e.g., the amount ofencouragement). If the person taps the face in the image, then moreextra resources may be applied to a facial recognition process than ifthe user taps the person's shoes in the image. In this latter case, ashoe identification process may be allocated a greater increase inresources than the facial recognition process. (Tapping the shoes canalso start a shoe recognition process, if not already underway.)

Another example of implied user encouragement is where the userpositions the camera so that a particular subject is at the center pointof the image frame. This is especially encouraging if the system notes atemporal sequence of frames, in which the camera is re-oriented—moving aparticular subject to the center point.

As before, the subject may be comprised of several parts (shoes,handbag, face, etc.). The distance between each such part, and thecenter of the frame, can be taken as inversely related to the amount ofencouragement. That is, the part at the center frame is impliedlyencouraged the most, with other parts encouraged successively less withdistance. (A mathematical function can relate distance to encouragement.For example, the part on which the frame is centered can have anencouragement value of 100, on a scale of 0 to 100. Any part at the farperiphery of the image frame can have an encouragement value of 0.Intermediate positions may correspond to encouragement values by alinear relationship, a power relationship, a trigonometric function, orotherwise.)

If the camera is equipped with a zoom lens (or digital zoom function),and the camera notes a temporal sequence of frames in which the camerais zoomed into a particular subject (or part), then such action can betaken as implied user encouragement for that particular subject/part.Even without a temporal sequence of frames, data indicating the degreeof zoom can be taken as a measure of the user's interest in the framedsubject, and can be mathematically transformed into an encouragementmeasure.

For example, if the camera has a zoom range of 1× to 5×, a zoom of 5×may correspond to an encouragement factor of 100, and a zoom of 1× maycorrespond to an encouragement factor of 1. Intermediate zoom values maycorrespond to encouragement factors by a linear relationship, a powerrelationship, a trigonometric function, etc.

Inference of intent may also be based on the orientation of featureswithin the image frame. Users are believed to generally hold imagingdevices in an orientation that frames intended subjects vertically. Byreference to accelerometer or gyroscope data, or otherwise, the devicecan discern whether the user is holding the imager in position tocapture a “landscape” or “portrait” mode image, from which “vertical”can be determined. An object within the image frame that has a principalaxis (e.g., an axis of rough symmetry) oriented vertically is morelikely to be a subject of the user's intention than an object that isinclined from vertical.

(Other clues for inferring the subject of a user's intent in an imageframe are discussed in U.S. Pat. No. 6,947,571.)

While the preceding discussion contemplated non-negative encouragementvalues, in other embodiments negative values can be utilized, e.g., inconnection with express or implied user disinterest in particularstimuli, remoteness of an image feature from the center of the frame,etc.

Encouragement—of both positive and negative varieties—can be provided byother processes. If a bar code detector starts sensing that the objectat the center of the frame is a bar code, its detection state metricincreases. Such a conclusion, however, tends to refute the possibilitythat the subject at the center of the frame is a face. Thus, an increasein detection state metric by a first recognition agent can serve asnegative encouragement for other recognition agents that are likelymutually exclusive with that first agent.

The encouragement and detection state metrics for plural recognitionagents can be combined by various mathematical algorithms to yield ahybrid control metric. One is their sum—yielding an output ranging from0-200 in the case of two agents (absent negative values forencouragement). Another is their product, yielding an output rangingfrom 0-10,000. Resources can be re-allocated to different recognitionagents as their respective hybrid control metrics change.

The recognition agents can be of different granularity and function,depending on application. For example, the facial recognition processjust-discussed may be a single pie slice of many stages. Or it can beimplemented as several, or dozens, of related, simpler processes—eachits own slice.

It will be recognized that the pie slice recognition agents in FIG. 12are akin to DLLs—code that is selectively loaded/invoked to provide adesired class of services. (Indeed, in some implementations, softwareconstructs associated with DLLs can be used, e.g., in the operatingsystem to administer loading/unloading of agent code, to publish theavailability of such functionality to other software, etc. DLL-basedservices can also be used in conjunction with recognition agents.)However, the preferred recognition agents have behavior different thanDLLs. In one aspect, this different behavior may be described asthrottling, or state-hopping. That is, their execution—and supportingresources—vary based on one or more factors, e.g., detection state,encouragement, etc.

FIG. 13 shows another view of the FIG. 12 arrangement. This viewclarifies that different processes may consume differing amounts ofprocessor time and/or other resources. (Implementation, of course, canbe on a single processor system, or a multi-processor system. In thefuture, different processors or “cores” of a multi-processor system maybe assigned to perform different of the tasks.)

Sometimes a recognition agent fails to achieve its goal(s) for lack ofsatisfactory resources, whether processing resources, input data, orotherwise. With additional or better resources, the goal might beachieved.

For example, a facial recognition agent may fail to recognize the faceof a person depicted in imagery because the camera was inclined 45degrees when the image was captured. At that angle, the nose is notabove the mouth—a criterion the agent may have applied in discerningwhether a face is present. With more processing resources, thatcriterion might be relaxed or eliminated. Alternatively, the face mighthave been detected if results from another agent—e.g., an orientationagent—had been available, e.g., identifying the inclination of the truehorizon in the imagery. Knowing the inclination of the horizon couldhave allowed the facial recognition agent to understand “above” in adifferent way—one that would have allowed it to identify a face.(Similarly, if a previously- or later-captured frame was analyzed, aface might have been discerned.)

In some arrangements the system does further analysis on input stimuli(e.g., imagery) when other resources become available. To cite a simplecase, when the user puts the phone into a purse, and the camera sensorgoes dark or hopelessly out of focus (or when the user puts the phone ona table so it stares at a fixed scene—perhaps the table or the ceiling),the software may reactivate agent processes that failed to achieve theiraim earlier, and reconsider the data Without the distraction ofprocessing a barrage of incoming moving imagery, and associated resourceburdens, these agents may now be able to achieve their original aim,e.g., recognizing a face that was earlier missed. In doing this, thesystem may recall output data from other agent processes—both thoseavailable at the time the subject agent was originally running, and alsothose results that were not available until after the subject agentterminated. This other data may aid the earlier-unsuccessful process inachieving its aim. (Collected “trash” collected during the phone'searlier operation may be reviewed for clues and helpful information thatwas overlooked—or not yet available—in the original processingenvironment in which the agent was run.) To reduce battery drain duringsuch an “after-the-fact mulling” operation, the phone may switch to apower-saving state, e.g., disabling certain processing circuits,reducing the processor clock speed, etc.

In a related arrangement, some or all of the processes that concluded onthe phone without achieving their aim may be continued in the cloud. Thephone may send state data for the unsuccessful agent process to thecloud, allowing the cloud processor to resume the analysis (e.g.,algorithm step and data) where the phone left off. The phone can alsoprovide the cloud with results from other agent processes—includingthose not available when the unsuccessful agent process was concluded.Again, data “trash” can also be provided to the cloud as a possibleresource, in case information earlier discarded takes on new relevancein the cloud's processing. The cloud can perform a gleaning operation onall such data—trying to find useful nuggets of information, or meaning,that the phone system may have overlooked. These results, when returnedto the phone, may in turn cause the phone to re-assess information itwas or is processing, perhaps allowing it to discern useful informationthat would otherwise have been missed. (E.g., in its data gleaningprocess, the cloud may discover that the horizon seems to be inclined 45degrees, allowing the phone's facial recognition agent to identify aface that would otherwise have been missed.)

While the foregoing discussion focused on recognition agents, the sametechniques can also be applied to other processes, e.g., those ancillaryto recognition, such as establishing orientation, or context, etc.

More on Constraints

FIG. 14 is a conceptual view depicting certain aspects of technologythat can be employed in certain embodiments. The top of the drawing showa hopper full of recognition agent (RA) services that could be run—mostassociated with one or more keyvectors to be used as input for thatservice. However, system constraints do not permit execution of allthese services. Thus, the bottom of the hopper is shown graphically asgated by constraints—allowing more or less services to be initiateddepending on battery state, other demands on CPU, etc.

Those services that are allowed to run are shown under the hopper. Asthey execute they may post interim or final results to the blackboard.(In some embodiments they may provide outputs to other processes or datastructures, such as to a UI manager, to another recognition agent, to anaudit trail or other data store, to signal to the operating system—e.g.,for advancing a state machine, etc.)

Some services run to completion and terminate (shown in the drawing bysingle strike-through)—freeing resources that allow other services to berun. Other services are killed prior to completion (shown by doublestrike-through). This can occur for various reasons. For example,interim results from the service may not be promising (e.g., an oval nowseems more likely a car tire than a face). Or system constraints maychange—e.g., requiring termination of certain services for lack ofresources. Or other, more promising, services may become ready to run,requiring reallocation of resources. Although not depicted in the FIG.14 illustration, interim results from processes that are killed may beposted to the blackboard—either during their operation, or at the pointthey are killed. (E.g., although a facial recognition application mayterminate if an oval looks more like a car tire than a face, a vehiclerecognition agent can use such information.)

Data posted to the blackboard is used in various ways. One is to triggerscreen display of baubles, or to serve other user interfacerequirements.

Data from the blackboard may also be made available as input torecognition agent services, e.g., as an input keyvector. Additionally,blackboard data may signal a reason for a new service to run. Forexample, detection of an oval—as reported on the blackboard—may signalthat a facial recognition service should be run. Blackboard data mayalso increase the relevance score of a service already waiting in the(conceptual) hopper—making it more likely that the service will be run.(E.g., an indication that the oval is actually a car tire may increasethe relevance score of a vehicle recognition process to the point thatthe agent process is run.)

The relevance score concept is shown in FIG. 15. A data structuremaintains a list of possible services to be run (akin to the hopper ofFIG. 14). A relevance score is shown for each. This is a relativeindication of the importance of executing that service (e.g., on a scaleof 1-100). The score can be a function of multiple variables—dependingon the particular service and application, including data found on theblackboard, context, expressed user intent, user history, etc. Therelevance score typically changes with time as more data becomesavailable, the context changes, etc. An on-going process can update therelevance scores based on current conditions.

Some services may score as highly relevant, yet require more systemresources than can be provided, and so do not run. Other services mayscore as only weakly relevant, yet may be so modest in resourceconsumption that they can be run regardless of their low relevancescore. (In this class may be the regularly performed image processingoperations detailed earlier.)

Data indicating the cost to run the service—in terms of resourcerequirements, is provided in the illustrated data structure (under theheading Cost Score in FIG. 15). This data allows a relevance-to-costanalysis to be performed.

The illustrated cost score is an array of plural numbers—eachcorresponding to a particular resource requirement, e.g., memory usage,CPU usage, GPU usage, bandwidth, other cost (such as for those servicesassociated with a financial charge), etc. Again, an arbitrary 0-100score is shown in the illustrative arrangement. Only three numbers areshown (memory usage, CPU usage, and cloud bandwidth), but more or lesscould of course be used.

The relevance-to-cost analysis can be as simple or complex as the systemwarrants. A simple analysis is to subtract the combined cost componentsfrom the relevance score, e.g., yielding a result of −70 for the firstentry in the data structure. Another simple analysis is to divide therelevance by the aggregate cost components, e.g., yielding a result of0.396 for the first entry.

Similar calculations can be performed for all services in the queue, toyield net scores by which an ordering of services can be determined. Anet score column is provided in FIG. 15, based on the first analysisabove.

In a simple embodiment, services are initiated until a resource budgetgranted to the Intuitive Computing Platform is reached. The Platformmay, for example, be granted 300 MB of RAM memory, a data channel of 256Kbits/second to the cloud, a power consumption of 50 milliwatts, andsimilarly defined budgets for CPU, GPU, and/or other constrainedresources. (These allocations may be set by the device operating system,and change as other system functions are invoked or terminate.) When anyof these thresholds is reached, no more recognition agent services arestarted until circumstances change.

While simple, this arrangement caps all services when a first of thedefined resource budgets is reached. Generally preferable arearrangements that seek to optimize the invoked services in view ofseveral or all of the relevant constraints. Thus, if the 256 Kbit/secondcloud bandwidth constraint is reached, then the system may stillinitiate further services that have no need for cloud bandwidth.

In more sophisticated arrangements, each candidate service is assigned afigure of merit score for each of the different cost componentsassociated with that service. This can be done by the subtraction ordivision approaches noted above for calculation of the net score, orotherwise. Using the subtraction approach, the cost score of 37 formemory usage of the first-listed service in FIG. 15 yields a memoryfigure of merit of 9 (i.e., 46-37). The service's figures of merit forCPU usage and cloud bandwidth are −18 and 31, respectively. By scoringthe candidate services in terms of their different resourcerequirements, a selection of services can be made that more efficientlyutilizes system resources.

As new recognition agents are launched and others terminate, and othersystem processes vary, the resource headroom (constraints) will change.These dynamic constraints are tracked (FIG. 16), and influence theprocess of launching (or terminating) recognition agents. If amemory-intensive RA completes its operation and frees 40 MB of memory,the Platform may launch one or more other memory-intensive applicationsto take advantage of the recently-freed resource.

(The artisan will recognize that the task of optimizing consumption ofdifferent resources by selection of different services is an exercise inlinear programming, to which there are many well known approaches. Thearrangements detailed here are simpler than those that may be employedin practice, but help illustrate the concepts.)

Returning to FIG. 15, the illustrated data structure also includes“Conditions” data A service may be highly relevant, and resources may beadequate to run it. However, conditions precedent to the execution maynot yet be met. For example, another Registration Agent service thatprovides necessary data may not yet have completed. Or the user (oragent software) may not yet have approved an expenditure required by theservice, or agreed to a service's click-wrap legal agreement, etc.

Once a service begins execution, there can be a programmed bias to allowit to run to completion, even if resource constraints change to put theaggregate Intuitive Computing Platform above its maximum budget.Different biases can be associated with different services, and withdifferent resources for a given service. FIG. 15 shows biases fordifferent constraints, e.g., memory, CPU and cloud bandwidth. In somecases, the bias may be less than 100%, in which case the service wouldnot be launched if availability of that resource is below the biasfigure.

For example, one service may continue to run until the aggregate ICPbandwidth is at 110% of its maximum value, whereas another service mayterminate immediately when the 100% threshold is crossed.

If a service is a low user of a particular resource, a higher bias maybe permitted. Or if a service has a high relevance score, a higher biasmay be permitted. (The bias may be mathematically derived from therelevance score, such as Bias=90+Relevance Score, or 100, whichever isgreater.)

Such arrangement allows curtailment of services in a programmable mannerwhen resource demands dictate, depending on biases assigned to thedifferent services and different constraints.

In some arrangements, services may be allowed to run, but withthrottled-back resources. For example, a service may normally have abandwidth requirement of 50 Kbit/sec. However, in a particularcircumstance, its execution may be limited to use of 40 Kbit/sec. Again,this is an exercise in optimization, the details of which will vary withapplication.

Local Software

In one particular embodiment, the local software on the mobile devicemay be conceptualized as performing six different classes of functions(not including installation and registering itself with the operatingsystem).

A first class of functions relates to communicating with the user. Thisallows the user to provide input, specifying, e.g., who the user is,what the user is interested in, what recognition operations are relevantto the user (tree leaves: yes; vehicle types: no), etc. (The user maysubscribe to different recognition engines, depending on interests.) Theuser interface functionality also provides the needed support for thehardware UI devices—sensing input on a touchscreen and keyboard,outputting information on the display screen etc.

To communicate effectively with the user, the software desirably hassome 3D understanding of the user's environment, e.g., how to organizethe 2D information presented on the screen, informed by knowledge thatthere's a 3D universe that is being represented; and how to understandthe 2D information captured by the camera, knowing that it represents a3D world. This can include a library of orthographic blittingprimitives. This gets into the second class.

A second class of functions relates to general orientation, orthographyand object scene parsing. These capabilities provide contextual commondenominators that can help inform object recognition operations (e.g.,the sky is up, the horizon in this image is inclined 20 degrees to theright, etc.)

A third class gets into actual pixel processing, and may be termedkeyvector Processing and Packaging. This is the universe of known pixelprocessing operations—transformations, template matching, etc., etc.Take pixels and crunch.

While 8×8 blocks of pixels are familiar in many image processingoperations (e.g., JPEG), that grouping is less dominant in the presentcontext (although it may be used in certain situations). Instead, fivetypes of pixel groupings prevail.

The first grouping is not a grouping at all, but global. E.g., is thelens cap on? What is the general state of focus? This is a categorywithout much—if any—parsing.

The second grouping is rectangular areas. A rectangular block of pixelsmay be requested for any number of operations.

The third grouping is non-rectangular contiguous areas.

Fourth is an enumerated patchworks of pixels. While still within asingle frame, this is a combination of the second and thirdgroupings—often with some notion of coherence (e.g., some metric or someheuristic that indicates a relationship between the included pixels,such as relevance to a particular recognition task).

Fifth is an interframe collections of pixels. These comprise a temporalsequence of pixel data (often not frames). As with the others, theparticular form will vary widely depending on application.

Another aspect of this pixel processing class of functions acknowledgesthat resources are finite, and should be allocated in increasing amountsto processes that appear to be progressing towards achieving their aim,e.g., of recognizing a face, and vice versa.

A fourth class of functions to be performed by the local software isContext Metadata Processing. This includes gathering a great variety ofinformation, e.g., input by the user, provided by a sensor, or recalledfrom a memory.

One formal definition of “context” is “any information that can be usedto characterize the situation of an entity (a person, place or objectthat is considered relevant to the interaction between a user and anapplication, including the user and applications themselves.”

Context information can be of many sorts, including the computingcontext (network connectivity, memory availability, CPU contention,etc.), user context (user profile, location, actions, preferences,nearby friends, social network(s) and situation, etc.), physical context(e.g., lighting, noise level, traffic, etc.), temporal context (time ofday, day, month, season, etc.), history of the above, etc.

A fifth class of functions for the local software is Cloud SessionManagement. The software needs to register different cloud-based serviceproviders as the resources for executing particular tasks, instantiateduplex sessions with the cloud (establishing IP connections, managingtraffic flow), ping remote service providers (e.g., alerting that theirservices may be required shortly), etc.

A sixth and final class of functions for the local software isRecognition Agent Management. These include arrangements for recognitionagents and service providers to publish—to cell phones—their inputrequirements, the common library functions on which they rely that mustbe loaded (or unloaded) at run-time, their data and other dependencieswith other system components/processes, their abilities to performcommon denominator processes (possibly replacing other serviceproviders), information about their maximum usages of system resources,details about their respective stages of operations (c.f., discussion ofFIG. 12) and the resource demands posed by each, data about theirperformance/behavior with throttled-down resources, etc. This sixthclass of functions then manages the recognition agents, given theseparameters, based on current circumstances, e.g., throttling respectiveservices up or down in intensity, depending on results and currentsystem parameters. That is, the Recognition Agent Management softwareserves as the means by which operation of the agents is mediated inaccordance with system resource constraints.

Sample Vision Applications

One illustrative application serves to view coins on a surface, andcompute their total value. The system applies an oval-finding process(e.g., a Hough algorithm) to locate coins. The coins may over-lie eachother and some may be only partially visible; the algorithm candetermine the center of each section of an oval it detects—eachcorresponding to a different coin. The axes of the ovals shouldgenerally be parallel (assuming an oblique view, i.e., that not all thecoins are depicted as circles in the imagery)—this can serve as a checkon the procedure.

After ovals are located, the diameters of the coins are assessed toidentify their respective values. (The assessed diameters can behistogrammed to ensure that they cluster at expected diameters, or atexpected diameter ratios.)

If a variety of several coins is present, the coins may be identified bythe ratio of diameters alone—without reference to color or indicia. Thediameter of a dime is 17.91 mm, the diameter of a penny is 19.05 mm; thediameter of a nickel is 21.21 mm; the diameter of a quarter is 24.26 mmRelative to the dime, the penny, nickel and quarter have diameter ratiosof 1.06, 1.18 and 1.35. Relative to the penny, the nickel and quarterhave diameter ratios of 1.11 and 1.27. Relative to the nickel, thequarter has a diameter ratio of 1.14.

These ratios are all unique, and are spaced widely enough to permitready discernment. If two coins have a diameter ratio of 1.14, thesmaller must be a nickel, the other must be a quarter. If two coins havea diameter ratio of 1.06, the smallest must be a dime, and the other apenny, etc. If other ratios are found, then something is amiss. (Notethat the ratio of diameters can be determined even if the coins aredepicted as ovals, since the dimensions of ovals viewed from the sameperspective are similarly proportional.)

If all of the coins are of the same type, they may be identified byexposed indicia.

In some embodiments, color can also be used (e.g., to aid indistinguishing pennies from dimes).

By summing the values of the identified quarters, with the values of theidentified dimes, with the values of the identified nickels, with thevalues of the identified pennies, the total value of coins on thesurface is determined. This value can be presented, or annunciated, tothe user through a suitable user interface arrangement.

A related application views a pile of coins and determines their countryof origin. The different coins of each country have a unique set ofinter-coin dimensional ratios. Thus, determination of diameter ratios—asabove—can indicate whether a collection of coins is from the US orCanada, etc. (The penny, nickel, dime, quarter, and half dollar ofCanada, for example, have diameters of 19.05 mm, 21.2 mm, 18.03 mm,23.88 mm, and 27.13 mm, so there is some ambiguity if the pile containsonly nickels and pennies, but this is resolved if other coins areincluded).

Augmented Environments

In many image processing applications, the visual context is welldefined. For example, a process control camera in a plywood plant may beviewing wood veneer on a conveyor belt under known lighting, or an ATMcamera may be grabbing security images of persons eighteen inches away,withdrawing cash.

The cell phone environment is more difficult—little or nothing may beknown about what the camera is viewing. In such instances it can bedesirable to introduce into the environment a known visiblefeature—something to give the system a visual toehold.

In one particular arrangement, machine vision understanding of a sceneis aided by positioning one or more features or objects in the field ofview for which reference information is known (e.g., size, position,angle, color), and by which the system can understand other features—byrelation. In one particular arrangement, target patterns are included inthe scene from which, e.g., the distance to, and orientation of,surfaces within the viewing space can be discerned. Such targets thusserve as beacons, signaling distance and orientation information to acamera system. One such target is the TRIPcode, detailed, e.g., in deIpiña, TRIP: a Low-Cost Vision-Based Location System for UbiquitousComputing, Personal and Ubiquitous Computing, Vol. 6, No. 3, May, 2002,pp. 206-219.

As detailed in the Ipiña paper, the target (shown in FIG. 17) encodesinformation including the target's radius, allowing a camera-equippedsystem to determine both the distance from the camera to the target, andthe target's 3D pose. If the target is positioned on a surface in theviewing space (e.g., on a wall), the Ipiña arrangement allows acamera-equipped system to understand both the distance to the wall, andthe wall's spatial orientation relative to the camera.

The TRIPcode has undergone various implementations, being successivelyknown as SpotCode, and then ShotCode (and sometimes Bango). It is nowunderstood to be commercialized by OP3 B.V.

The aesthetics of the TRIPcode target are not suited for someapplications, but are well suited for others. For example, carpet orrugs may be fashioned incorporating the TRIPcode target as a recurrentdesign feature, e.g., positioned at regular or irregular positionsacross a carpet's width. A camera viewing a scene that includes a personstanding on such a carpet can refer to the target in determining thedistance to the person (and also to define the plane encompassing thefloor). In like fashion, the target can be incorporated into designs forother materials, such as wallpaper, fabric coverings for furniture,clothing, etc.

In other arrangements, the TRIPcode target is made less conspicuous byprinting it with an ink that is not visible to the human visual system,but is visible, e.g., in the infrared spectrum. Many image sensors usedin mobile phones are sensitive well into the infrared spectrum. Suchtargets may thus be discerned from captured image data, even though thetargets escape human attention.

In still further arrangements, the presence of a TRIPcode can becamouflaged among other scene features, in manners that nonethelesspermit its detection by a mobile phone.

One camouflage method relies on the periodic sampling of the image sceneby the camera sensor. Such sampling can introduce visual artifacts incamera-captured imagery (e.g., aliasing, Moiré effects) that are notapparent when an item is inspected directly by a human. An object can beprinted with a pattern designed to induce a TRIPcode target to appearthrough such artifact effects when imaged by the regularly-spacedphotosensor cells of an image sensor, but is not otherwise apparent tohuman viewers. (This same principle is advantageously used in makingchecks resistant to photocopy-based counterfeiting. A latent image, suchas the word VOID, is incorporated into the graphical elements of theoriginal document design. This latent image isn't apparent to humanviewers. However, when sampled by the imaging system of a photocopier,the periodic sampling causes the word VOID to emerge and appear inphotocopies.) A variety of such techniques are detailed in van Renesse,Hidden and Scrambled Images—a Review, Conference on Optical Security andCounterfeit Deterrence Techniques IV, SPIE Vol. 4677, pp. 333-348, 2002.

Another camouflage method relies on the fact that color printing iscommonly performed with four inks: cyan, magenta, yellow and black(CMYK). Normally, black material is printed with black ink. However,black can also be imitated by overprinting cyan and magenta and yellow.To humans, these two techniques are essentially indistinguishable. To adigital camera, however, they may readily be discerned. This is becauseblack inks typically absorb a relatively high amount of infrared light,whereas cyan, magenta and yellow channels do not.

In a region that is to appear black, the printing process can apply(e.g., on a white substrate) an area of overlapping cyan, magenta andyellow inks. This area can then be further overprinted (or pre-printed)with a TRIPcode, using black ink. To human viewers, it all appearsblack. However, the camera can tell the difference, from the infraredbehavior. That is, at a point in the black-inked region of the TRIPcode,there is black ink obscuring the white substrate, which absorbs anyincident infrared illumination that might otherwise be reflected fromthe white substrate. At another point, e.g., outside the TRIPcodetarget, or inside its periphery—but where white normally appears—theinfrared illumination passes through the cyan, magenta and yellow inks,and is reflected back to the sensor from the white substrate.

The red sensors in the camera are most responsive to infraredillumination, so it is in the red channel that the TRIPcode target isdistinguished. The camera may provide infrared illumination (e.g., byone or more IR LEDs), or ambient lighting may provide sufficient IRillumination. (In future mobile devices, a second image sensor may beprovided, e.g., with sensors especially adapted for infrared detection.)

The arrangement just described can be adapted for use with any colorprinted imagery—not just black regions. Details for doing so areprovided in patent application 20060008112. By such arrangement,TRIPcode targets can be concealed wherever printing may appear in avisual scene, allowing accurate mensuration of certain features andobjects within the scene by reference to such targets.

While a round target, such as the TRIPcode, is desirable forcomputational ease, e.g., in recognizing such shape in its differentelliptical poses, markers of other shapes can be used. A square markersuitable for determining the 3D position of a surface is Sony'sCyberCode and is detailed, e.g., in Rekimoto, CyberCode: DesigningAugmented Reality Environments with Visual Tags, Proc. of DesigningAugmented Reality Environments 2000, pp. 1-10. A variety of otherreference markers can alternatively be used—depending on therequirements of a particular application. One that is advantageous incertain applications is detailed in published patent application20100092079 to Aller.

In some arrangements, a TRIPcode (or CyberCode) can be further processedto convey digital watermark data. This can be done by the CMYKarrangement discussed above and detailed in the noted patentapplication. Other arrangements for marking such machine-readable datacarriers with steganographic digital watermark data, and applicationsfor such arrangements, are detailed in U.S. Pat. No. 7,152,786 andpatent application 20010037455.

Another technology that can be employed with similar effect are Bokodes,as developed at MIT's Media Lab. Bokodes exploit the bokeh effect ofcamera lenses—mapping rays exiting from an out of focus scene point intoa disk-like blur on the camera sensor. An off the shelf camera cancapture Bokode features as small as 2.5 microns from a distance of 10feet or more. Binary coding can be employed to estimate the relativedistance and angle to the camera. This technology is further detailed inMohan, Bokode: Imperceptible Visual Tags for Camera Based Interactionfrom a Distance, Proc. of SIGGRAPH'09, 28(3):1-8.

Multi-Touch Input, Image Re-Mapping, and Other Image Processing

As noted elsewhere, users may tap proto-baubles to express interest inthe feature or information that the system is processing. The user'sinput raises the priority of the process, e.g., by indicating that thesystem should apply additional resources to that effort. Such a tap canlead to faster maturation of the proto-bauble into a bauble.

Tapping baubles can also serve other purposes. For example, baubles maybe targets of touches for user interface purposes in a manner akin tothat popularized by the Apple iPhone (i.e., its multi-touch UI).

Previous image multi-touch interfaces dealt with an image as anundifferentiated whole. Zooming, etc., was accomplished without regardto features depicted in the image.

In accordance with a further aspect of the present technology,multi-touch and other touch screen user interfaces perform operationsthat are dependent, in part, on some knowledge about what one or moreparts of the displayed imagery represent.

To take a simple example, consider an oblique-angle view of severalitems scattered across the surface of a desk. One may be a coin—depictedas an oval in the image frame.

The mobile device applies various object recognition steps as detailedearlier, including identifying edges and regions of the imagecorresponding to potentially different objects. Baubles may appear.Tapping the location of the coin in the image (or a bauble associatedwith the coin), the user can signal to the device that the image is tobe re-mapped so that the coin is presented as a circle—as if in a planview looking down on the desk. (This is sometimes termedortho-rectification.)

To do this, the system desirably first knows that the shape is a circle.Such knowledge can derive from several alternative sources. For example,the user may expressly indicate this information (e.g., through theUI—such as by tapping the coin and then tapping a circle controlpresented at a margin of the image, indicating the tapped object iscircular in true shape). Or such a coin may be locally recognized by thedevice—e.g., by reference to its color and indicia (or cloud processingmay provide such recognition). Or the device may assume that anysegmented image feature having the shape of an oval is actually a circleviewed from an oblique perspective. (Some objects may include machinereadable encoding that can be sensed—even obliquely—and indicate thenative shape of the object. For example, QR bar code data may bediscerned from a rectangular object, indicating the object's true shapeis a square.) Etc.

Tapping on the coin's depiction in the image (or a corresponding bauble)may—without more—cause the image to be remapped. In other embodiments,however, such instruction requires one or more further directions fromthe user. For example, the user's tap may cause the device to present amenu (e.g., graphical or auditory) detailing several alternativeoperations that can be performed. One can be plan re-mapping.

In response to such instruction, the system enlarges the scale of thecaptured image along the dimension of the oval's minor axis, so that thelength of that minor axis equals that of the oval's major axis.(Alternatively, the image can be shrunk along the major axis, withsimilar effect.) In so doing, the system has re-mapped the depictedobject to be closer to its plan view shape, with the rest of the imageremapped as well.

In another arrangement, instead of applying a scaling factor to just onedirection, the image may be scaled along two different directions. Insome embodiments, shearing can be used, or differential scaling (e.g.,to address perspective effect).

A memory can store a set of rules by which inferences about an object'splan shape from oblique views can be determined. For example, if anobject has four approximately straight sides, it may be assumed to be arectangle—even if opposing sides are not parallel in the camera's view.If the object has no apparent extent in a third dimension, is largelyuniform in a light color—perhaps with some high frequency dark markingsamid the light color, the object may be assumed to be a piece ofpaper—probably with an 8.5:11 aspect ratio if GPS indicates a locationin the US (or 1:SQRT(2) if GPS indicates a location in Europe). There-mapping can employ such information—in the lack of other knowledge—toeffect a view transformation of the depicted object to somethingapproximating a plan view.

In some arrangements, knowledge about one segmented object in the imageframe can be used to inform or refine a conclusion about another objectin the same frame. Consider an image frame depicting a round object thatis 30 pixels in its largest dimension, and another object that is 150pixels in its largest dimension. The latter object may be identified—bysome processing—to be a coffee cup. A data store of referenceinformation indicates that coffee cups are typically 3-6″ in theirlongest dimension. Then the former object can be deduced to have adimension on the order of an inch (not, e.g., a foot or a meter, asmight be the case of round objects depicted in other images).

More than just size classification can be inferred in this manner. Forexample, a data store can include information that groups associateditems together. Tire and car. Sky and tree. Keyboard and mouse. Shavingcream and razor. Salt and pepper shakers (sometimes with ketchup andmustard dispensers). Coins and keys and cell phone and wallet. Etc.

Such associations can be gleaned from a variety of sources. One istextual metadata from image archives such as Flickr or Google Images(e.g., identify all images with razor in the descriptive metadata,collect all other terms from such images' metadata, and rank in terms ofoccurrence, e.g., keeping the top 25%). Another is by natural languageprocessing, e.g., by conducting a forward-linking analysis of one ormore texts (e.g., a dictionary and an encyclopedia), augmented bydiscerning inverse semantic relationships, as detailed in U.S. Pat. No.7,383,169.

Dimensional knowledge can be deduced in similar ways. For example, aseed collection of reference data can be input to the data store (e.g.,a keyboard is about 12-20″ in its longest dimension, a telephone isabout 8-12,″ a car is about 200,″ etc.). Images can then be collectedfrom Flickr including the known items, together with others. Forexample, Flickr presently has nearly 200,000 images tagged with the term“keyboard.” Of those, over 300 also are tagged with the term “coffeecup.” Analysis of similar non-keyboard shapes in these 300+ imagesreveals that the added object has a longest dimension roughly a thirdthat of the longest dimension of the keyboard. (By similar analysis, amachine learning process can deduce that the shape of a coffee cup isgenerally cylindrical, and such information can also be added to theknowledge base—local or remote—consulted by the device.)

Inferences like those discussed above typically do not render a finalobject identification. However, they make certain identifications morelikely (or less likely) than others, and are thus useful, e.g., inprobabilistic classifiers.

Sometimes re-mapping of an image can be based on more than the imageitself. For example, the image may be one of a sequence of images, e.g.,from a video. The other images may be from other perspectives, allowinga 3D model of the scene to be created. Likewise if the device has stereoimagers, a 3D model can be formed. Re-mapping can proceed by referenceto such a 3D model.

Similarly, by reference to geolocation data, other imagery from the samegeneral location may be identified (e.g., from Flickr, etc.), and usedto create a 3D model, or to otherwise inform the re-mapping operation.(Likewise, if Photosynths continue to gain in popularity andavailability, they provide rich data from which remapping can proceed.)

Such remapping is a helpful step that can be applied to captured imagerybefore recognition algorithms, such as OCR, are applied. Consider, forexample, the desk photo of the earlier example, also depicting atelephone inclined up from the desk, with an LCD screen displaying aphone number. Due to the phone's inclination and the viewing angle, thedisplay does not appear as a rectangle but as a rhomboid. Recognizingthe quadrilateral shape, the device may re-map it into a rectangle(e.g., by applying a shear transformation). OCR can then proceed on there-mapped image—recognizing the characters displayed+on the telephonescreen.

Returning to multi-touch user interfaces, additional operations can beinitiated by touching two or more features displayed on the devicescreen.

Some effect other remapping operations. Consider the earlier deskexample, depicting both a telephone/LCD display inclined up from thedesk surface, and also a business card lying flat. Due to theinclination of the phone display relative to the desk, these twotext-bearing features lie in different planes. OCRing both from a singleimage requires a compromise.

If the user touches both segmented features (or baubles corresponding toboth), the device assesses the geometry of the selected features. Itthen computes, for the phone, the direction of a vector extending normalto the apparent plane of the LCD display, and likewise for a vectorextending normal from the surface of the business card. These twovectors can then be averaged to yield an intermediate vector direction.The image frame can then be remapped so that the computed intermediatevector extends straight up. In this case, the image has been transformedto yield a plan view onto a plane that is angled midway between theplane of the LCD display and the plane of the business card. Such aremapped image presentation is believed to be the optimum compromise forOCRing text from two subjects lying in different planes (assuming thetext on each is of similar size in the remapped image depiction).

Similar image transformations can be based on three or more featuresselected from an image using a multi-touch interface.

Consider a user at a historical site, with interpretative signage allaround. The signs are in different planes. The user's device captures aframe of imagery depicting three signs, and identifies the signs asdiscrete objects of potential interest from their edges and/or otherfeatures. The user touches all three signs on the display (orcorresponding baubles, together or sequentially). Using a procedure likethat just-described, the planes of the three signs are determined, and acompromise viewing perspective is then created to which the image isremapped—viewing the scene from a direction perpendicular to an averagesignage plane.

Instead of presenting the three signs from the compromise viewingperspective, an alternative approach is to remap each sign separately,so that it appears in plan view. This can be done by converting thesingle image to three different images—each with a different remapping.Or the pixels comprising the different signs can be differently-remappedwithin the same image frame (warping nearby imagery to accommodate thereshaped, probably enlarged, sign depictions).

In still another arrangement, touching the three signs (at the sametime, or sequentially) initiates an operation that involves obtainingother images of the designated objects from an image archive, such asFlickr or Photosynth. (The user may interact with a UI on the device tomake the user's intentions clear, e.g., “Augment with other pixel datafrom Flickr.”) These other images may be identified by pose similaritywith the captured image (e.g., lat/long, plus orientation), or otherwise(e.g., other metadata correspondence, pattern matching, etc.). Higherresolution, or sharper-focused, images of the signs may be processedfrom these other sources. These sign excerpts can be scaled andlevel-shifted as appropriate, and then blended and pasted into the imageframe captured by the user—perhaps processed as detailed above (e.g.,remapped to a compromise image plane, remapped separately—perhaps in 3different images, or in a composite photo warped to accommodate thereshaped sign excerpts, etc.).

In the arrangements just detailed, analysis of shadows visible in thecaptured image allows the device to gain certain 3D knowledge about thescene (e.g., depth and pose of objects) from a single frame. Thisknowledge can help inform any of the operations detailed above.

Just as remapping an image (or excerpt) can aid in OCRing, it can alsoaid in deciding what other recognition agent(s) should be launched.

Tapping on two features (or baubles) in an image can initiate a processto determine a spatial relationship between depicted objects. In acamera view of a NASCAR race, baubles may overlay different race cars,and track their movement. By tapping baubles for adjoining cars (ortapping the depicted cars themselves), the device may obtain locationdata for each of the cars. This can be determined in relative terms fromthe viewer's perspective, e.g., by deducing locations of the cars fromtheir scale and position in the image frame (knowing details of thecamera optics and true sizes of the cars). Or the device can link to oneor more web resources that track the cars' real time geolocations, e.g.,from which the user device can report that the gap between the cars iseight inches and closing.

(As in earlier examples, this particular operation may be selected froma menu of several possible operations when the user taps the screen.)

Instead of simply tapping baubles, a further innovation concernsdragging one or more baubles on the screen. They can be dragged ontoeach other, or onto a region of the screen, by which the user signals adesired action or query.

In an image with several faces, the user may drag two of thecorresponding baubles onto a third. This may indicate a groupingoperation, e.g., that the indicated people have some socialrelationship. (Further details about the relationship may be input bythe user using text input, or by spoken text—through speechrecognition.) In a network graph sense, a link is established betweendata objects representing the two individuals. This relationship caninfluence how other device processing operations deal with the indicatedindividuals.

Alternatively, all three baubles may be dragged to a new location in theimage frame. This new location can denote an operation, or attribute, tobe associated with the grouping—either inferentially (e.g., context), orexpressed by user input.

Another interactive use of feature-proxy baubles is in editing an image.Consider an image with three faces: two friends and a stranger. The usermay want to post the image to an online repository (Facebook) but maywant to remove the stranger first. Baubles can be manipulated to thisend.

Adobe Photoshop CS4 introduced a feature termed Smart Scaling, which waspreviously known from online sites such as rsizr<dot>com. Areas ofimagery that are to be saved are denoted (e.g., with a mouse-drawnbounding box), and other areas (e.g., with superfluous features) arethen shrunk or deleted. Image processing algorithms preserve the savedareas unaltered, and blend them with edited regions that formerly hadthe superfluous features.

In the present system, after processing a frame of imagery to generatebaubles corresponding to discerned features, the user can execute aseries of gestures indicating that one feature (e.g., the stranger) isto be deleted, and that two other features (e.g., the two friends) areto be preserved. For example, the user may touch the unwanted bauble,and sweep the finger to the bottom edge of the display screen toindicate that the corresponding visual feature should be removed fromthe image. (The bauble may follow the finger, or not). The user may thendouble-tap each of the friend baubles to indicate that they are to bepreserved. Another gesture calls up a menu from which the user indicatesthat all the editing gestures have been entered. The processor thenedits the image according to the user's instructions. An “undo” gesture(e.g., a counterclockwise half-circle finger trace on the screen) canreverse the edit if it proved unsatisfactory, and the user may tryanother edit. (The system may be placed in a mode to receive editingbauble gestures by an on-screen gesture, e.g., finger-tracing the letter‘e,’ or by selection from a menu, or otherwise.)

The order of a sequence of bauble-taps can convey information about theuser's intention to the system, and elicit corresponding processing.

Consider a tourist in a new town, viewing a sign introducing variouspoints of interest, with a photo of each attraction (e.g., Eiffel Tower,Arc de Triomphe, Louvre, etc). The user's device may recognize some orall of the photos, and present a bauble corresponding to each depictedattraction. Touching the baubles in a particular order may instruct thedevice to obtain walking directions to the tapped attractions, in theorder tapped. Or it may cause the device to fetch Wikipedia entries foreach of the attractions, and present them in the denoted order.

Since feature-proxy baubles are associated with particular objects, orimage features, they can have a response—when tapped or included in agesture—dependent on the object/feature to which they correspond. Thatis, the response to a gesture can be a function of metadata associatedwith the baubles involved.

For example, tapping on a bauble corresponding to a person can signifysomething different (or summon a different menu of available operations)than tapping on a bauble corresponding to a statue, or a restaurant.(E.g., a tap on the former may elicit display or annunciation of theperson's name and social profile, e.g., from Facebook; a tap on thesecond may summon Wikipedia information about the statue or itssculptor; a tap on the latter may yield the restaurant's menu, andinformation about any current promotions.) Likewise, a gesture thatinvolves taps on two or more baubles can also have a meaning thatdepends on what the tapped baubles represent, and optionally the orderin which they were tapped.

Over time, a gesture vocabulary that is generally consistent acrossdifferent baubles may become standardized. Tapping once, for example,may summon introductory information of a particular type correspondingto the type of bauble (e.g., name and profile, if a bauble associatedwith a person is tapped; address and directory of offices, if a baubleassociated with a building is tapped; a Wikipedia page, if a bauble fora historical site is tapped; product information, if a bauble for aretail product is tapped, etc.). Tapping twice may summon a highlightsmenu of, e.g., the four most frequently invoked operations, againtailored to the corresponding object/feature. A touch to a bauble, and awiggle of the finger at that location, may initiate anotherresponse—such as display of an unabridged menu of choices, with a scrollbar. Another wiggle may cause the menu to retract.

Notes on Architecture

This specification details a number of features. Althoughimplementations can be realized with a subset of features, they aresomewhat less preferred. Reasons for implementing a richer, rather thansparser, set of features, are set forth in the following discussion.

An exemplary software framework supports visual utility applicationsthat run on a smartphone, using a variety of components:

1. The screen is a real-time modified camera image, overlaid by dynamicicons (baubles) that can attach to portions of the image and actsimultaneously as value displays and control points for (possible)multiple actions occurring at once. The screen is also a valuable,monetizable advertising space (in a manner similar to Google's searchpages)—right at the focus of the user's attention.

2. Many applications for the device process live sequences of cameraimages, not mere “snapshots.” In many cases, complex image judgments arerequired, although responsiveness remains a priority.

3. The actual applications will ordinarily be associated with displayedbaubles and the currently visible “scene” shown by the display—allowinguser interaction to be a normal part of all levels of theseapplications.

4. A basic set of image-feature extraction functions can run in thebackground, allowing features of the visible scene to be available toapplications at all times.

5. Individual applications desirably are not permitted to “hog” systemresources, since the usefulness of many will wax and wane with changesin the visible scene, so more than one application will often be activeat once. (This generally requires multitasking, with suitable dispatchcapabilities, to keep applications lively enough to be useful.)

6. Applications can be designed in layers, with relatively low-loadfunctions which can monitor the scene data or the user desires, withmore intensive functions invoked when appropriate. The dispatcharrangements can support this code structure.

7. Many applications may include cloud-based portions to performoperations beyond the practical capabilities of the device itself.Again, the dispatch arrangements can support this capability.

8. Applications often require a method (e.g., the blackboard) to postand access data which is mutually useful.

In a loose, unordered way, below are some of the interrelationships thatcan make the above aspects parts of a whole—not just individuallydesirable.

1. Applications that refer to live scenes will commonly rely onefficient extraction of basic image features, from all (or at leastmany) frames—so making real-time features available is an importantconsideration (even though, for certain applications, it may not berequired).

2. In order to allow efficient application development and testing, aswell as to support applications on devices with varying capabilities, anability to optionally place significant portions of any application “inthe cloud” will become nearly mandatory. Many benefits accrue from suchcapability.

3. Many applications will benefit from recognition capabilities that arebeyond the current capabilities of unaided software. These applicationswill demand interaction with a user to be effective. Further, mobiledevices generally invite user interactions—and only if the GUI supportsthis requirement will consistent, friendly interaction be possible.

4. Supporting complex applications on devices with limited, inflexibleresources requires full support from the software architecture.Shoehorning PC-style applications onto these devices is not generallysatisfactory without careful redesign. Multitasking of layered softwarecan be an important component of providing an inviting user experiencein this device-constrained environment.

5. Providing image information to multiple applications in an efficientmanner is best done by producing information only once, and allowing itsuse by every application that needs it—in a way that minimizesinformation access and caching inefficiencies. The “blackboard” datastructure is one way of achieving this efficiency.

Thus, while aspects of the detailed technology are useful individually,it is in combination that their highest utility may be realized.

More on Blackboard

Garbage collection techniques can be employed in the blackboard toremove data that is no longer relevant. Removed data may be transferredto a long term store, such as a disk file, to serve as a resource inother analyses. (It may also be transferred, or copied, to the cloud—asnoted elsewhere.)

In one particular arrangement, image- and audio-based keyvector data isremoved from the blackboard when a first of alternate criteria is met,e.g., a new discovery session begins, or the user's location changes bymore than a threshold (e.g., 100 feet or 1000 feet), or a stalenessperiod elapses (e.g., 3, or 30, or 300 seconds) since the keyvector datawas generated. In the former two cases, the old data may be retainedfor, e.g., N further increments of time (e.g., 20 further seconds) afterthe new discovery session begins, or M further increments (e.g., 30further seconds) after the user's location changes by more than thethreshold.

Non-image/audio keyvector data (e.g., accelerometer, gyroscope, GPS,temperature) are typically kept on the blackboard longer thanimage/audio keyvector data, in view of their limited storagerequirements. For example, such data may persist on the blackboard untilthe phone next is in a sleep (low battery drain) state of operation formore than four hours, or until several such successive sleep states haveoccurred.

If any aging blackboard data is newly utilized (e.g., used as input by arecognition agent, or newly found to relate to other data), itspermitted residency on the blackboard is extended. In one particulararrangement it is extended by a time period equal to the period from thedata's original creation until its new utilization (e.g., treating itsnew utilization time as a new creation time). Keyvector data relating toa common object may be aggregated together in a new keyvector form,similarly extending its permitted blackboard lifetime.

Data can also be restored to the blackboard after its removal (e.g.,from a long-term store), if the removed data was gathered within athreshold measure of geographical proximity to the user's currentposition. For example, if the blackboard was populated withimage-related keyvector data while the user was at a shopping mall, andthe user drove back home (flushing the blackboard), then when the usernext returns to that mall, the most-recently flushed keyvector datacorresponding to that location can be restored to the blackboard. (Theamount of data restored is dependent on the blackboard size, andavailability.)

In some respects, the blackboard may be implemented, or another datastructure may serve, as a sort of automated Wild for objects, focused onsensor fusion. Every few seconds (or fractions of a second), pages ofdata are shed, and links between data elements are broken (or new onesare established). Recognition agents can populate pages and set uplinks. Pages are frequently edited—with the state machine commonlyserving as the editor. Each Wild author can see every other page, andcan contribute.

The system may also invoke trust procedures, e.g., in connection withthe blackboard. Each time a recognition agent tries to newly post datato the blackboard, it may be investigated in a trust system database todetermine its reliability. The database can also indicate whether theagent is commercial or not. Its ratings by users can be considered indetermining a reliability score to be given to its data (or whetherparticipation with the blackboard should be permitted at all). Based ontrust findings and stored policy data, agents can be granted or refusedcertain privileges, such as contributing links, breaking links (its own,or that of third parties), deleting data (its own, or that of thirdparties), etc.

In one particular arrangement, a device may consult with an independenttrust authority, such as Verisign or TRUSTe, to investigate arecognition agent's trustworthiness. Known cryptographic techniques,such as digital signature technology, can be employed to authenticatethat third party providing the agent service is who it claims to be, andthat any agent software is untampered-with. Only if such authenticationsucceeds, and/or only if the independent trust authority rates theprovider with a grade above a threshold (e.g., “B,” or 93 out of 100,which may be user-set) is the recognition agent granted the privilege ofinteracting with the device's blackboard structure (e.g., by readingand/or writing information).

The device may similarly investigate the privacy practices of serviceproviders (e.g., through TRUSTe) and allow interaction only if certainthresholds are exceeded, or parameters are met.

More on Processing, Usage Models, Compass, and Sessions

As noted, some implementations capture imagery on a free-running basis.If limited battery power is a constraint (as is presently the usualcase), the system may process this continuing flow of imagery in ahighly selective mode in certain embodiments—rarely applying asignificant part (e.g., 10% or 50%) of the device's computationalcapabilities to analysis of the data. Instead, it operates in a lowpower consumption state, e.g., performing operations without significantpower cost, and/or examining only a few frames each second or minute (ofthe, e.g., 15, 24 or 30 frames that may be captured every second). Onlyif (A) initial, low level processing indicates a high probability thatan object depicted in the imagery can be accurately recognized, and (B)context indicates a high probability that recognition of such objectwould be relevant to the user, does the system throttle up into a secondmode in which power consumption is increased. In this second mode, thepower consumption may be more than two-times, or 10-, 100-, 1000- ormore-times the power consumption in the first mode. (The notedprobabilities can be based on calculated numeric scores dependent on theparticular implementation. Only if these scores—for successful objectrecognition, and for relevance to the user—exceed respective thresholdvalues (or combine per a formula to exceed a single threshold value),does the system switch into the second mode.) Of course, if the usersignals interest or encouragement, expressly or impliedly, or if contextdictates, then the system can also switch out of the first mode into thesecond mode.

The emerging usage model for certain augmented reality (AR)applications, e.g., in which a user is expected to walk the streets of acity while holding out a smart phone and concentrating on its changingdisplay (e.g., to navigate to a desired coffee shop or subway station),is ill-advised. Numerous alternatives seem preferable.

One is to provide guidance audibly, through an earpiece or a speaker.Rather than providing spoken guidance, more subtle auditory clues can beutilized—allowing the user to better attend to other auditory input,such as car horns or speech of a companion. One auditory clue can beoccasional tones or clicks that change in repetition rate or frequencyto signal whether the user is walking in the correct direction, andgetting closer to the intended destination. If the user tries to make awrong turn at an intersection, or moves away-from rather than towardsthe destination, the pattern can change in a distinctive fashion. Oneparticular arrangement employs a Geiger counter-like sound effect, witha sparse pattern of clicks that grows more frequent as the userprogresses towards the intended destination, and falls off if the userturns away from the correct direction. (In one particular embodiment,the volume of the auditory feedback changes in accordance with usermotion. If the user is paused, e.g., at a traffic light, the volume maybe increased—allowing the user to face different directions andidentify, by audio feedback, in which direction to proceed. Once theuser resumes walking, the audio volume can diminish, until the user onceagain pauses. Volume, or other user feedback intensity level, can thusdecrease when the user is making progress per the navigation directions,and increase when the user pauses or diverts from the expected path.)

Motion can be detected in various ways, such as by accelerometer orgyroscope output, by changing GPS coordinates, by changing scenerysensed by the camera, etc.

Instead of auditory feedback, the above arrangements can employvibratory feedback instead.

The magnetometer in the mobile device can be used in theseimplementations to sense direction. However, the mobile device may beoriented in an arbitrary fashion relative to the user, and the user'sdirection of forward travel. If it is clipped to the belt of anorth-facing user, the magnetometer may indicate the device is pointingto the north, or south, or any other direction—dependent on the how thedevice is oriented on the belt.

To address this issue, the device can discern a correction factor to beapplied to the magnetometer output, so as to correctly indicate thedirection the user is facing. For example, the device can sense adirectional vector along which the user is moving, by reference tooccasional GPS measurements. If, in ten seconds, the user's GPScoordinates have increased in latitude, but stayed constant inlongitude, then the user has moved north—presumably while facing in anortherly direction. The device can note the magnetometer output duringthis period. If the device is oriented in such a fashion that itsmagnetometer has been indicating “east,” while the user has apparentlybeen facing north, then a correction factor of 90 degrees can bediscerned. Thereafter, the device knows to subtract ninety degrees fromthe magnetometer-indicated direction to determine the direction the useris facing—until such an analysis indicates a different correction shouldbe applied. (Such technique is broadly applicable—and is not limited tothe particular arrangement detailed here.)

Of course, such methods are applicable not just to walking, but also tobicycling and other modes of transportation.

While the detailed arrangements assumed that imagery is analyzed as itis captured, and that the capturing is performed by the user device,neither is required. The same processing may be performed on imagery (oraudio) captured earlier and/or elsewhere. For example, a user's devicemay process imagery captured an hour or week ago, e.g., by a publiccamera in a city parking lot. Other sources of imagery include Flickrand other such public image repositories, YouTube and other video sites,imagery collected by crawling the public web, etc.

(It is advantageous to design the processing software so that it caninterchangeably handle both live and canned image data, e.g., live imagestills or streams, and previously recorded data files. This allowsseemingly different user applications to employ the same inner core. Tosoftware designers, this is also useful as it allows live-imageapplications to be repeatedly tested with known images or sequences.)

Many people prefer to review voice mails in transcribed textform—skimming for relevant content, rather than listening to everyutterance of a rambling talker. In like fashion, results based on asequence of visual imagery can be reviewed and comprehended by manyusers more quickly than the time it took to capture the sequence.

Consider a next generation mobile device, incorporating aheadwear-mounted camera, worn by a user walking down a city block.During the span of the block, the camera system may collect 20, 60 ormore seconds of video. Instead of distractedly (while walking) viewingan overlaid AR presentation giving results based on the imagery, theuser can focus on the immediate tasks of dodging pedestrians andobstacles. Meanwhile, the system can analyze the captured imagery andstore the result information for later review. (Or, instead of capturingimagery while walking, the user may pause, sweep a camera-equipped smartphone to capture a panorama of imagery, and then put the phone back in apocket or purse.)

(The result information can be of any form, e.g., identification ofobjects in the imagery, audio/video/text information obtained relatingto such objects, data about other action taken in response to visualstimuli, etc.)

At a convenient moment, the user can glance at a smart phone screen (oractivate a heads-up display on eyewear) to review results produced basedon the captured sequence of frames. Such review can involve presentationof response information alone, and/or can include the captured imageryon which the respective responses were based. (In cases where responsesare based on objects, an object may appear in several frames of thesequence. However, the response need only be presented for one of theseframes.) Review of the results can be directed by the device, in astandardized presentation, or can be directed by the user. In the lattercase, the user can employ a UI control to navigate through the resultsdata (which may be presented in association with image data, or not).One UI is the familiar touch interface popularized by the Apple iPhonefamily. For example, the user can sweep through a sequence of scenes(e.g., frames captured 1 or 5 seconds, or minutes, apart), each withoverlaid baubles that can be tapped to present additional information.Another navigation control is a graphical or physical shuttlecontrol—familiar from video editing products such as AdobePremier—allowing the user to speed forward, pause, or reverse thesequence of images and/or responses. Some or all of the resultinformation may be presented in auditory form, rather than visual. Theuser interface can be voice-responsive, rather than responsive, e.g., totouch.

While the visual information was collected in a video fashion, the usermay find it most informative to review the information in static scenefashion. These static frames are commonly selected by the user, but maybe selected, or pre-filtered, by the device, e.g., omitting frames thatare of low quality (e.g., blurry, or occluded by an obstacle in theforeground, or not having much information content).

The navigation of device-obtained responses need not traverse the entiresequence (e.g., displaying each image frame, or each response). Somemodalities may skip ahead through the information, e.g., presenting onlyresponses (and/or images) corresponding to every second frame, or everytenth, or some other interval of frame count or time. Or the review canskip ahead based on saliency, or content. For example, parts of asequence without any identified feature or corresponding response may beskipped entirely. Images with one or a few identified features (or otherresponse data) may be presented for a short interval. Images with manyidentified features (or other response data) may be presented for alonger interval. The user interface may present a control by which theuser can set the overall pace of the review, e.g., so that a sequencethat took 30 seconds to capture may be reviewed in ten seconds, or 20,or 30 or 60, etc.

It will be recognized that the just-described mapping of review-time tocapture-time may be non-linear, such as due to time-varying saliency ofthe imagery (e.g., some excerpts are rich in interesting objects; othersare not), etc. For example, if a sequence that is reviewed in 15 secondstook 60 seconds to capture, then one-third through the review may notcorrespond to one-third through the capture, etc. So subjects may occurat time locations in the review data that are non-proportional to theirtime-locations in the capture data.

The user interface can also provide a control by which the user canpause any review, to allow further study or interaction, or to requestthe device to further analyze and report on a particular depictedfeature. The response information may be reviewed in an ordercorresponding to the order in which the imagery was captured, or reverseorder (most recent first), or can be ordered based on estimatedrelevance to the user, or in some other non-chronological fashion.

Such interactions, and analysis, may be regarded as employing asession-based construct. The user can start the review in the middle ofthe image sequence, and traverse it forwards or backwards, continuously,or jumping around. One of the advantages to such a session arrangementis that later-acquired imagery can help inform understanding ofearlier-acquired imagery. To cite but one example, a person's face maybe revealed in frame 10 (and recognized using facial recognitiontechniques), whereas only the back of the person's head may be shown inframe 5. Yet by analyzing the imagery as a collection, the person can becorrectly labeled in frame 5, and other understanding of the frame 5scene can be based on such knowledge. In contrast, if scene analysis isbased exclusively on the present and preceding frames, the person wouldbe anonymous in frame 5.

Session constructs can be used through the embodiments detailed herein.Some sessions have natural beginning and/or ending points. For example,abrupt scene transformations in captured video can serve to start or enda session, as when a user takes a camera out of a pocket to scan ascene, and later restores it to the pocket. (Techniques borrowed fromMPEG can be employed for this purpose, e.g., detecting a scene changethat requires start of a new Group of Pictures (GOP)—beginning with an“I” frame.) A scene losing its novelty can be used to end a session,just as a scene taking on new interest can start one. (E.g., if a camerahas been staring out in space from a bedside table overnight, and isthen picked up—newly introducing motion into the imagery, this cantrigger the start of a session. Conversely, if the camera is left in afixed orientation in a static environment, this lack of new visualstimulus can soon cause a session to end.)

Audio analogs to image-based sessions can alternatively, oradditionally, be employed.

Other sensors in the phone can also be used to trigger the start or endof a session, such as accelerometers or gyroscopes signaling that theuser has picked up the phone or changed its orientation.

User action can also expressly signal the start, or end of a session.For example, a user may verbally instruct a device to “LOOK AT TONY.”Such a directive is an event that serves as a logical start of a newsession. (Directives may be issued other than by speech, e.g., byinteraction with a user interface, by shaking a phone to signal that itscomputational resources should be focused/increased on stimulusthen-present in the environment, etc.)

Some sessions may be expressly invoked, by words such as DISCOVER orSTART. These sessions may terminate in response to a signal from asoftware timer (e.g., after 10, 30, 120, 600 seconds—depending on storedconfiguration data), unless earlier stopped by a directive, such as STOPor QUIT. A UI warning that the timer is approaching the end of thesession may be issued to the user, and a selection of buttons or othercontrol arrangements can be presented—allowing extension of the sessionfor, e.g., 10, 30, 120 or 600 seconds, or indefinitely (or allowing theuser to enter another value).

To avoid unnecessary data capture, and instructional ambiguity,directives such as “JUST LOOK” or “JUST LISTEN” may be issued by a user.In the former case, no audio data is sampled (or, if sampled, it is notstored). Reciprocally with the latter.

Similarly, the user may state “LISTEN TO THE MUSIC” or “LISTEN TO THESPEECH.” In each case, captured data can be segmented and identified asto class, and analysis can focus on the designated type. (The other maybe discarded.)

Likewise, the user may state “LISTEN TO TV.” In addition to otherprocessing that this instruction may invoke, it also clues the processorto look for digital watermark data of the sort encoded by The NielsenCompany in television audio. (Such watermark is encoded in a particularspectral range, e.g., 2 KHz-5 KHz. With knowledge of such information,the device can tailor its sampling, filtering and analysis accordingly.)

Sometimes data extraneous to an intended discovery activity is captured.For example, if the length of a session is set by a timer, or determinedby a period of visual inactivity (e.g., ten seconds), then the sessionmay capture information—particularly near the end—that has no value forthe intended discovery operation. The system can employ a process toidentify what data is relevant to the intended discovery operation, anddiscard the rest. (Or, similarly, the system can identify what data isnot relevant to the intended discovery operation, and discard it.)

Consider a user in an electronics store, who is capturing imagery ofproducts of potential interest—particularly their barcodes. The sessionmay also capture audio and other imagery, e.g., of store patrons. Fromthe video data, and particularly its movement to successive barcodes—onwhich the user dwells, the system can infer that the user is interestedin product information. In such case it may discard audio data, andvideo not containing barcodes. (Likewise, it may discard keyvector datanot relating to barcodes.) In some implementations the system checkswith the user before undertaking such action, e.g., detailing itshypothesis of what the user is interested in, and asking forconfirmation. Only keyvector data corresponding to barcode regions ofimagery may be retained.

While session usually denotes a temporal construct, e.g., an intervalthat encompasses a series of logically related events or processes,other session constructs can also be employed. For example, a logicalsession may be defined by reference to a particular spatial regionwithin an image frame, or within an image sequence (in which case theregion may exhibit motion). (MPEG-4 objects may each be regarded interms of spatial sessions. Likewise with other object-oriented datarepresentations.)

It should be recognized that plural sessions can be ongoing at a time,overlapping in whole or part, beginning and ending independently. Orplural sessions may share a common start (or end), while they end (orstart) independently. A shake of (or tap on) a phone, for example, maycause the phone to pay increased attention to incoming sensor data. Thephone may respond by applying increased processing resources tomicrophone and camera data. The phone may quickly discern, however, thatthere is no microphone data of note, whereas the visual scene ischanging dramatically. It may thus terminate an audio processing sessionafter a few seconds—reducing resources applied to analysis of the audio,while continuing a video processing session much longer, e.g., until theactivity subsides, a user action signals a stop, etc.

As noted earlier data from discovery sessions is commonly stored, andcan be recalled later. In some instances, however, a user may wish todiscard the results of a session. A UI control can allow such an option.

Verbal directives, such as “LOOK AT TONY,” can greatly assist devices intheir operation. In some arrangements a phone needn't be on heightenedalert all the time—trying to discern something useful in a never-endingtorrent of sensor data. Instead, the phone can normally be in a loweractivity state (e.g., performing processing at a background levelestablished by stored throttle data), and commit additional processingresources only as indicated.

Such directive also serves as an important clue that can shortcut otherprocessing. By reference to a stored data (e.g., in a local or remotedatabase), the phone can quickly recognize that “Tony” is a member ofone or more logical classes, such as human, person, male, FaceBookfriend, and/or face. The phone can launch or tailor processes to discernand analyze features associated with such a class entity. Put anotherway, the phone can identify certain tasks, or classes of objects, withwhich it needn't be concerned. (“LOOK AT TONY” can be regarded as adirective not to look for a banknote, not to decode a barcode, not toperform song recognition, not to focus on a car, etc., etc. Thoseprocesses may be terminated if underway, or simply not started duringthe session.) The directive thus vastly reduces the visual search spacewith which the device must cope.

The stored data consulted by the phone in interpreting the user'sdirective can be of various forms. One is a simple glossary thatindicates, for each word or phrase, one or more associated descriptors(e.g., “person,” “place” or “thing;” or one or more other classdescriptors). Another is the user's phone book—listing names, andoptionally providing images, of contacts. Another is the user's socialnetworking data, e.g., identifying friends and subjects of interest.Some such resources can be in the cloud—shared across groups of users.In some cases, such as the phone book, the stored data can include imageinformation—or clues—to assist the phone in its imageprocessing/recognition task.

Voice recognition technology useful in such embodiment is familiar tothe artisan. Accuracy of the recognition can be increased by limitingthe universe of candidate words between which the recognition algorithmmust match. By limiting the glossary to a thousand (or a hundred, orfewer) words, extremely high recognition accuracy can be achieved withlimited processing, and with limited time. (Such an abridged glossarymay include friends' names, common instructional words such as START,STOP, LOOK, LISTEN, YES, NO, GO, QUIT, END, DISCOVER, common colors,digits and other numbers, popular geographic terms in the current area,etc.) Google's speech recognition technology used in its GOOG411 productcan be employed if speed (or local data storage) isn't a paramountconcern. Related information on speech recognition technologies isdetailed in the present assignee's application 20080086311.

Directives from the user needn't be familiar words with establisheddefinitions. They can be utterances, snorts, nasal vocalizations,grunts, or other sounds made by the user in certain contexts. “UH-UNH”can be taken as a negative—indicating to the phone that its currentfocus or results are not satisfactory. “UM-HMM” can be taken as anaffirmation—confirming that the phone's processing is in accord with theuser's intent. The phone can be trained to respond appropriately to suchutterances, as with other unrecognized words.

Directives needn't be auditory. They can be otherwise, such as bygesture. Again, the phone can ascribe meanings to gestures throughtraining experiences.

In some embodiments, visual projections can direct the phone to asubject of interest. For example, a user can point to a subject ofinterest using a laser pointer having a known spectral color, or adistinctive temporal or spectral modulation. A microprojector cansimilarly be utilized to project a distinctive target (e.g., that ofFIG. 17, or a 3×3 array of spots) onto an object of interest—usingvisible light or infrared. (If visible light is used, the target can beprojected infrequently, e.g., for a thirtieth of a second eachsecond—timing to which detection software may be synced. If infrared, itmay be projected with a red laser pointer dot to show the user where aninfrared pattern is placed. In some cases, the targets may beindividualized, e.g., serialized, to different users, to allow thesimultaneous presence of many projected targets, such as in a publicspace.) Such projected target not only indicates the subject ofinterest, but also allows orientation of, and distance to, the object tobe determined (its pose)—establishing “ground truth” useful in otheranalyses. Once the projected feature is found within the imagery, thesystem can segment/analyze the image to identify the object on which thetarget is found, or take other responsive action.

In some arrangements, the phone is always looking for such projecteddirectives. In others, such action is triggered by the user verballyinstructing “LOOK FOR LASER” or “LOOK FOR TARGET.” This is an examplewhere a combination of directives is employed: spoken and visuallyprojected. Other combinations of different types of directives are alsopossible.

If the system doesn't recognize a particular directive, or fails in itsattempt to complete an associated task, it can indicate same by feedbackto the user, such as by a raspberry sound, an audio question (e.g.,“who?” or “what?”), by a visual message, etc.

For example, the phone may understand that “LOOK AT TONY” is a directiveto process imagery to discern a friend of the user (for whom referenceimagery may be available in storage). However, because of the phonecamera's perspective, it may not be able to recognize Tony within thefield of view (e.g., his back may be to the camera), and may indicatethe failure to the user. The user may respond by trying otherdirectives, such as “HAT,” “GREEN SHIRT,” “NEAR,” “GUY ON RIGHT,”etc.—other clues by which the intended subject or action can beidentified.

A user in a mall may capture imagery showing three items on a shelf. Byspeaking “THE MIDDLE ONE,” the user may focus the phone's processingresources on learning about the object in the middle, to the exclusionof objects on the right and left (and elsewhere). Other descriptors canlikewise be used (e.g., “IT'S THE RED ONE,” or “THE SQUARE ONE,” etc.)

From such examples, it will be recognized that audio clues (and/or otherclues) can be used as a means of bounding an ICP device's processingefforts. Object recognition is thus supplemented/aided by speechrecognition (and/or other clues).

(Conversely, speech recognition can be supplemented/aided by objectrecognition. For example, if the device recognizes that the user'sfriend Helen is in the camera's field of view, and if a word of spokenspeech is ambiguous—it might be “hero” or “Helen” or “hello”—thenrecognizing the person Helen in imagery may tip resolution of theambiguity to “Helen.” Similarly, if the visual context indicates a pondwith ducks, an ambiguous word might be resolved as “fowl,” whereas ifthe visual context indicates a baseball stadium, the same word might beresolved as “foul.”) Location data, such as from GPS, can similarly beused in resolving ambiguities in speech. (If the location data indicatesthe user is at a Starbucks (such as through one of the known servicesthat associates descriptors with latitude/longitude data), an ambiguousutterance might be resolved as “tea,” whereas on a golf course, the sameutterance might be resolved as “tee.”)

The system's response to speech can vary, depending on what processingthe phone is undertaking, or has completed. For example, if the phonehas analyzed a street scene, and overlaid visual baubles correspondingto different shops and restaurants, then the user speaking the name ofone of these shops or restaurants may be taken as equivalent to tappingthe displayed bauble. If a bar called “The Duck” has a bauble on thescreen, then speaking the name “DUCK” may cause the phone to display thebar's happy hour menu. In contrast, if on a hike, a user's phone hasrecognized a Mallard duck in a pond, and the user speaks “DUCK,” thismay summon display of the Wikipedia page for Mallard ducks. Stillfurther, if in November, the phone recognizes the University of Oregon“O” logo on a car window and overlays a corresponding bauble on theuser's phone screen, then speaking the word “DUCK” may summon a rosteror game schedule for the Oregon Ducks football team. (If it's February,the same circumstances may summon a roster or game schedule for theOregon Ducks basketball team.) Thus, different responses to the samespoken word(s) may be provided, depending on processing the phone hasundertaken (and/or varying with indicia displayed on the phone screen).

As just noted, responses may also differ depending on location, time ofday, or other factor(s). At mid-day, speaking the name of a restaurantfor which a bauble is displayed may summon the restaurant's lunch menu.In the evening, the dinner menu may be displayed instead. Speaking thename “HILTON,” when a Hilton hotel is nearby, can display the room ratesfor the nearby property. (The same “HILTON” word prompts displays ofdifferent room rates in Detroit than in New York City.)

Speaking to a phone allows a conversational mode of instruction. Inresponse to an initial instruction, the phone can undertake an initialset of operations. Seeing the actions undertaken responsive to theinitial instruction (or results therefrom), the user can issue furtherinstructions. The phone, in turn, responds with further operations. Inan iterative fashion, the user can interactively guide the phone toproduce the user-desired results. At any point, the user can direct thatthe session be saved, so that the iterative process can be resumed at alater time. While “saved,” processing can continue, e.g., in the cloud,so that when the user returns to the interaction at a later time,additional information may be available.

“Saving” can be implemented differently, based on user preference orapplication, and privacy considerations. In some cases, only a digest ofa session is preserved. A digest may include location data (e.g., fromGPS), direction/orientation data (e.g., from magnetometers), anddate/time. The originally captured image/audio may be retained, butoften is not. Instead, derivatives may be preserved. One type ofderivative is a content fingerprint—data derived from human-intelligiblecontent, but from which the human-intelligible content cannot bereconstructed. Another type of derivative is keyvector data, e g, dataidentifying shapes, words, SIFT data, and other features. Another typeof derivative data is decoded machine readable information, such aswatermark or barcode payloads. Derived data that identifies content,such as song titles and television program names, may also be preserved.

In some cases, originally captured image/audio data may bepreserved—provided permission is received from the person(s) that suchdata represents. Derivative data may also require permission forpreservation, if it is associated with a person (e.g., facialidentification vectors, voiceprint information).

Just as popular cameras draw rectangles around perceived faces in thecamera view-finder to indicate the subject on which the camera'sauto-focus and exposure will be based, an ICP device may draw arectangle, or provide other visual indicia, around a visual subjectpresented on the device screen to inform the user what in the imagery isto the focus of the device's processing.

In some embodiments, rather than directing the device's attention byspoken clues or instructions (or in addition thereto), the user cantouch an object as displayed on the screen, or circle it, to indicatethe subject on which the device should concentrate its effort. Thisfunctionality may be enabled even if the system has not yet displayed(or does not display) a bauble corresponding to the object.

Declarative Configuration of Sensor-Related Systems

This section further details some of the concepts noted above.

In the prior art, smart phones have used speech recognition for purposessuch as hands-free dialing, and for spoken internet queries (semanticsearch). In accordance with certain embodiments of the presenttechnology, speech recognition is employed in connection with tuning theoperation of one or more sensor-based systems, so as to enhanceextraction of information desired by the user.

Referring to FIG. 25, an exemplary smart phone 710 includes varioussensors, such as a microphone 712 and a camera 714, each with arespective interface 716, 718. Operation of the phone is controlled by aprocessor 720, configured by software instructions stored in a memory722.

The phone 710 is shown as including a speech recognition module 724.This functionality may be implemented by the phone's processor 720, inconjunction with associated instructions in memory 722. Or it can be adedicated hardware processor. In some embodiments, this functionalitymay be external to the phone—with data passed to and from an externalspeech recognition server through the phone's RF cellular- or datatransceiver-capabilities. Or the speech recognition functionality can bedistributed between the phone and a remote processor.

In use, a user speaks one or more words. The microphone 712 senses theassociated audio, and the interface electronics 716 convert analogsignals output by the microphone into digital data. This audio data isprovided to the speech recognition module 724, which returns recognizedspeech data.

The user may speak, for example, “LISTEN TO THE MAN.” The phone canrespond to this recognized speech instruction by applying a male voicefilter to audio sensed by the microphone. (The voiced speech of atypical male has fundamental frequencies down to about 85 Hertz, so thefilter may remove frequencies below that value.) If the user says“LISTEN TO THE WOMAN,” the phone may respond by applying a filteringfunction that removes frequencies below 165 Hz—the bottom range of atypical woman's voice. In both cases the filtering function applied bythe phone responsive to such instructions may cut out audio frequenciesabout 2500 or 3000 Hz—the upper end of the typical voice frequency band.(Audio filtering is sometimes termed “equalization,” and can involveboosting, as well as attenuating, different audio frequencies.)

The phone thus receives a spoken indication of a subject in the user'senvironment, in which the user is interested (e.g., “man”), andconfigures its signal processing of received audio accordingly. Such anarrangement is depicted in FIG. 26.

The configuration of the phone can be accomplished by establishingparameters used in connection with signal processing, such as samplingrates, filter cutoff frequencies, watermark key data, addresses ofdatabases to be consulted, etc. In other arrangements, the configurationcan be accomplished by executing different software instructionscorresponding to different signal processing operations. Or theconfiguration can be accomplished by activating different hardwareprocessing circuits, or routing data to external processors, etc.

In one particular implementation, the phone includes a table or otherdata structure that associates different spoken subjects (e.g., “man,”“woman,” “radio,” “television,” “song,” etc.) with different signalprocessing operations, as shown by the table excerpt of FIG. 27. Eachword recognized by the speech recognition engine is applied to thetable. If any recognized word matches one of the “subjects” identifiedin the table, the phone then applies the specified signal processinginstructions to audio thereafter received (e.g., in the currentsession). In the depicted example, if the phone recognizes “man,” itapplies a corresponding male voice filtering function to the audio, andpasses the filtered audio to the speech recognition engine. Text that isoutput from the speech recognition is then presented on the phone'sdisplay screen—per directions specified by the table.

The user may speak “LISTEN TO THE RADIO.” Consulting the table of FIG.27, the phone responds to this recognized speech data by attempting toidentify the audio by detecting an Arbitron digital watermark. The audiois first sampled at a 6 KHz sampling frequency. It is then filtered, anda decoding procedure corresponding to the Arbitron watermark is applied(e.g., per stored software instructions). The decoded watermark payloadis transmitted to Arbitron's remote watermark database, and metadatarelating to the radio broadcast is returned from the database to thehandset. The phone then presents this metadata on its screen.

If an Arbitron watermark is not found in the audio, the instructions inthe table specify an alternative set of operations. In particular, this“Else” condition instructs the phone to apply the operations associatedwith the subject “Song.”

The instructions associated with “Song” start with lowpass filtering theaudio at 4 KHz. (Earlier-captured audio data may be buffered in a memoryto allow for such re-processing of earlier-captured stimulus.) A Shazamsong identification fingerprint is then computed (using instructionsstored separately), and the resulting fingerprint data is transmitted toShazam's song identification database. Corresponding metadata is lookedup in this database and returned to the phone for display. If nometadata is found, the display indicates the audio is not recognized.

(It should be understood that the detailed signal processing operationsmay be performed on the phone, or by a remote processor (e.g., in the“cloud”), or in distributed fashion. It should further be understoodthat the signal processing operations shown in FIG. 27 are only a smallsubset of a large universe of signal processing operations—and sequencesof operations—that can be triggered based on user input. When parametersare not specified in the instructions detailed in the table, defaultvalues can be used, e.g., 8 KHz for sampling rate, 4 KHz for low passfiltering, etc.)

Some smart phones include two or more microphones. In such case thesignal processing instructions triggered by user input can involveconfiguring the microphone array, such as by controlling the phasing andamplitude contribution from each microphone into a combined audiostream. Or, the instructions can involve processing audio streams fromthe different microphones separately. This is useful, e.g., for soundlocalization or speaker identification. Additional signal conditioningoperations may be applied to improve extraction of the desired audiosignal. Through sensor fusion techniques, the location of the speakercan be estimated based on the camera and pose-estimation techniquesamong others. Once the source is identified, and with the presence ofmultiple microphones, beam-forming techniques may be utilized to isolatethe speaker. Over a series of samples, the audio environment thatrepresents the channel can be modeled and removed to further improverecovery of the speaker's voice.

Phones typically include sensors other than microphones. Cameras areubiquitous. Other sensors are also common (e.g., RFID and near fieldcommunication sensors, accelerometers, gyroscopes, magnetometers, etc.).User speech can similarly be employed to configure processing of suchother sensor data.

In some embodiments, this functionality might be triggered by the userspeaking a distinctive key word or expression such as “DIGIMARC LOOK” or“DIGIMARC LISTEN”—initiating the application and cueing the device thatthe words to follow are not mere dictation. (In other embodiments, adifferent cue can be provided—spoken or otherwise, such as gestural. Instill other embodiments, such cue can be omitted.)

For example, “DIGIMARC LOOK AT THE TELEVISION” may evoke a specialdictionary of commands to trigger a sequence of signal processingoperations such as setting a frame capture rate, applying certain colorfilters, etc. “DIGIMARC LOOK AT PERSON” may launch a procedure thatincludes color compensation for accurate flesh-tones, extraction offacial information, and application of the face information to a facialrecognition system.

Again, a table or other data structure can be used to associatecorresponding signal processing operations with different actions andobjects of interest. Among the different objects for which instructionsmay be indicated in the table are “newspaper,” “book,” “magazine,”“poster,” “text,” “printing,” “ticket,” “box,” “package,” “carton,”“wrapper,” “product,” “barcode,” “watermark,” “photograph,” “photo,”“person,” “man,” “boy,” “woman,” “girl,” “him,” “her,” “them,” “people,”“display,” “screen,” “monitor,” “video,” “movie,” “television,” “radio,”“iPhone,” “iPad,” “Kindle,” etc. Associated operations can includeapplying optical character recognition, digital watermark decoding,barcode reading, calculating image or video fingerprints, and subsidiaryimage processing operations and parameters, such as color compensation,frame rates, exposure times, focus, filtering, etc.

Additional verbiage may be utilized to help segment a visual scene withobject descriptors colors, shapes, or location (foreground, background,etc.) Across multiple samples, temporal descriptors can be utilized,such as blinking, flashing, additional motion descriptors can beapplied, such fast, or slow.

Devices that contain sensors enabling them to identify motion of thedevice add another layer of control words, those that state arelationship between the device and the desired object. Simple commandssuch as “track,” might indicate that the device should segment thevisual or auditory scene to include only those objects whosetrajectories approximate the motion of the device.

In more elaborate arrangements, the phone includes several such tables,e.g., Table 1 for audio stimulus, Table 2 for visual stimulus, etc. Thephone can decide which to use based on other terms and/or syntax in therecognized user speech.

For example, if the recognized user speech includes verbs such as“look,” “watch,” “view,” “see,” or “read,” this can signal to the phonethat visual stimulus is of interest to the user. If one of these wordsis detected in the user's speech, the phone can apply other words orsyntax from the user's recognized speech to Table 2. Conversely, if therecognized user speech includes verbs such as “listen” or “hear,” thisindicates that the user is interested in audible stimulus, and Table 1should be consulted.

By such rule-based arrangement, the phone responds differently to thetwo spoken phrases “DIGIMARC LOOK AT THE MAN” and “DIGIMARC LISTEN TOTHE MAN.” In the former case, Table 2 (corresponding to visual stimuluscaptured by the camera) is consulted. In the latter case, Table 1(corresponding to audible stimulus captured by the microphone) isconsulted. FIGS. 28 and 29 show examples of such systems.

(The artisan will understand that the described arrangement of tables isonly one way of many by which the detailed functionality can beachieved. The artisan will similarly recognize that a great variety ofverbs and other words—beyond those detailed above—can be interpreted asclues as to whether the user is interested in visual or auditorystimulus.)

Sometimes a spoken noun also reveals something about the type ofstimulus. In the phrase, “DIGIMARC LOOK AT THE MAGAZINE,” “Digimarc”evokes the special libraries and operations, “Look” connotes visualstimulus, and “magazine” tells something about the visual stimulus aswell, i.e., that it comprises static printed images and/or text (whichcould be distinguished by use of “Read” rather than “Look.” In contrast,in the phrase “DIGIMARC, LOOK AT THE TELEVISION,” the term “television”indicates that the content has a temporal aspect, so that capturingplural frames for analysis is appropriate.

It will be recognized that by associating different parameters and/orsignal processing operations with different key terms, the phone isessentially reconfigured by spoken user input. One moment it isconfigured as a radio watermark detector. The next it is configured as afacial recognition system. Etc. The sensor-related systems aredynamically tuned to serve the user's apparent interests. Moreover, theuser generally does not explicitly declare a function (e.g., “READ ABARCODE”) but rather identifies a subject (e.g., “LOOK AT THE PACKAGE”)and the phone infers a function desired (or a hierarchy of possiblefunctions), and alters operation of the phone system accordingly.

In some cases involving the same operation (e.g., digital watermarkdecoding), the details of the operation can vary depending on theparticular subject. For example, a digital watermark in a magazine istypically encoded using different encoding parameters than a digitalwatermark embedded in a newspaper, due to the differences between theinks, media, and printing techniques used. Thus, “DIGIMARC, LOOK AT THEMAGAZINE” and “DIGIMARC, LOOK AT THE NEWSPAPER” may both involve digitalwatermark decoding operations, but the former may utilize decodingparameters different than the latter (e.g., relevant color space,watermark scale, payload, etc.). (The “Digimarc” intro is omitted in theexamples that follow, but the artisan will understand that such cue cannonetheless be used.)

Different subjects may be associated with typical differentcamera-viewing distances. If the user instructs “LOOK AT THE MAGAZINE,”the phone may understand (e.g., from other information stored in thetable) that the subject will be about 8 inches away, and can instruct amechanical or electronic system to focus the camera system at thatdistance. If the user instructs “LOOK AT THE ELECTRONIC BILLBOARD,” incontrast, the camera may focus at a distance of 8 feet. The scale ofimage features the phone expects to discern can be similarlyestablished.

Sometimes the user's spoken instruction may include a negation, such as“not” or “no” or “ignore.”

Consider a phone that normally responds to user speech “LOOK AT THEPACKAGE,” by examining captured image data for a barcode. If found, thebarcode is decoded, the payload data is looked-up in a database, andresulting data is then presented on the screen. If no barcode is found,the phone resorts to an “Else” instruction in the stored data, e g,analyzing the captured image data for watermark data, and submitting anydecoded payload data to a watermark database to obtain related metadata,which is then displayed on the screen. (If no watermark is found, afurther “Else” instruction may cause the phone to examine the imageryfor likely text, and submit any such excerpts to an OCR engine. Resultsfrom the OCR engine are then presented on the screen.)

If the user states “LOOK AT THE PACKAGE; IGNORE THE BARCODE,” thisalters the normal instruction flow. In this case the phone does notattempt to decode barcode data from captured imagery. Instead, itproceeds directly to the first “Else” instruction, i.e., examiningimagery for watermark data.

Sometimes the user may not particularly identify a subject. Sometimesthe user may only offer a negation, e.g., “NO WATERMARK.” In such casethe phone can apply a prioritized hierarchy of content processingoperations to the stimulus data (e.g., per a stored listing)—skippingoperations that are indicated (or inferred) from the user's speech asbeing inapplicable.

Of course, spoken indication of a subject of interest may be understoodas a negation of other subjects of potential interest, or as a negationof different types of processing that might be applied to stimulus data.(E.g., “LOOK AT THE MAN” clues the phone that it need not examine theimagery for a digital watermark, or a barcode.)

It will thus be understood that the user's declaration helps the phone'sprocessing system decide what identification technologies and otherparameters to employ in order to best meet the user's probable desires.

Speech recognition software suitable for use with the present technologyis available from Nuance Communications, e.g., its SpeechMagic andNaturallySpeaking SDKs. Free speech recognition software (e.g.,available under open source licenses) includes the Sphinx family ofofferings, from Carnegie Mellon University. This includes Sphinx 4 (aJAVA implementation), and Pocket Sphinx (a simplified version optimizedfor use on ARM processors). Other free speech recognition softwareincludes Julius (by a consortium of Japanese universities cooperating inthe Interactive Speech Technology Consortium), ISIP (from MississippiState) and VoxForge (an open source speech corpus and acoustic model,usable with Sphinx, Julius and ISIP).

While described in the context of sensing user interests by reference tothe user's spoken speech, other types of user input can also beemployed. Gaze (eye) tracking arrangements can be employed to identify asubject at which the user is looking. Pointing motions, either by a handor a laser pointer, can likewise be sensed and used to identify subjectsof interest. A variety of such user inputs that do not involve a usertactilely interacting with the smart phone (e.g., by a keyboard or bytouch gestures) can be used. Such arrangements are generally depicted inFIG. 30.

In some embodiments, the signal processing applied by the phone can alsobe based, in part, on context information.

As discussed elsewhere, one definition of “context” is “any informationthat can be used to characterize the situation of an entity (a person,place or object that is considered relevant to the interaction between auser and an application, including the user and applicationsthemselves.” Context information can be of many sorts, including thecomputing context (network connectivity, memory availability, CPUcontention, etc.), user context (user profile, location, actions,preferences, nearby friends, social network(s) and situation, etc.),physical context (e.g., lighting, noise level, traffic, etc.), temporalcontext (time of day, day, month, season, etc.), content context(subject matter, actors, genre, etc.), history of the above, etc.

More on Vision Operations and Related Notions

Because of their ability to dynamically apportion the desired tasksamong on-device resources and “the cloud,” certain embodiments of thepresent technology are well suited for optimizing application responsein the context of limited memory and computational resources.

For complex tasks, such as confirming the denomination of a banknote,one could refer the entire task to the most time- or cost-effectiveprovider. If the user wants to recognize a U.S. banknote, and anexternal provider (e.g., bidder) is found that can do it, the high-leveltask can be performed in the cloud. For efficiency, the cloud serviceprovider can use image feature data extracted by subtasks performed onthe device—e.g., processing the image data to minimize the externalbandwidth required, or filtered to remove personally-identifiable orextraneous data. (This locally processed data can simultaneously also bemade available to other tasks—both local and remote.)

In some arrangements, the details of the external provider's processingaren't known to the local device, which is instructed only as to thetype and format of input data required, and the type/format of outputdata provided. In other arrangements, the provider publishes informationabout the particular algorithms/analyses applied in performing itsprocessing, so that the local device can consider same in making achoice between alternate providers.

To the extent that the computational model focuses on certain tasksalways being capable of being performed on the device, these basicoperations would be tailored to the type of likely cloud applicationsenvisioned for each device. For example, if applications will needimages with specific resolution, contrast, and coverage of a banknote orother document, matching capabilities will be required for the ‘imageacquire’ functions provided.

In general, top-down thinking provides some very specific low-levelfeatures and capabilities for a device to provide. At that point, thedesigner will brainstorm a bit. What more useful features orcapabilities do these suggest? Once a list of such generally usefulcapabilities has been compiled, a suite of basic operations can beselected and provision made to minimize memory and power requirements.

As an aside, Unix has long made use of “filter chains” that can minimizeintermediate storage. To perform a sequence of transformations,cascadable “filters” are provided for each step. For instance, supposethe transformation A->B is actually a sequence:

-   -   A|op1|op2|op3>B

If each step takes an item into a new item of the same or similar size,and assuming that A is still to be available at the end, the memoryrequirement is size(A)+size(B)+2 buffers, with each buffer typicallymuch smaller than the full object size, and de-allocated when theoperation completes. Complex local transformations, for instance, can beobtained by combining a few simple local operations in this way. Bothstorage and the number of operations performed can be reduced, savingtime, power or both.

At least some applications are naturally conceived with short imagesequences as input. A system design can support this idea by providing ashort, perhaps fixed length (e.g., three or four, or 40, frames) imagesequence buffer, which is the destination for every image acquisitionoperation. Varying application requirements can be supported byproviding a variety of ways of writing to the buffers: one or more newimages FIFO inserted; one or more new images combined via filters (min,max, average, . . . ) then FIFO inserted; one or more new imagescombined with the corresponding current buffer elements via filters theninserted, etc.

If an image sequence is represented by a fixed-size buffer, filled in aspecific fashion, extracting an image from a sequence would be replacedby extracting an image from the buffer. Each such extraction can selecta set of images from the buffer and combine them via filters to form theextracted image. After an extraction, the buffer may be unchanged, mayhave had one or more images removed, or may have some of its imagesupdated by a basic image operation.

There are at least three types of subregions of images that are commonlyused in pattern recognition. The most general is just a set of extractedpoints, with their geometric relationships intact, usually as a list ofpoints or row fragments. The next is a connected region of the image,perhaps as a list of successive row fragments. The last is a rectangularsub-image, perhaps as an array of pixel values and an offset within theimage.

Having settled on one or more of these feature types to support, arepresentation can be selected for efficiency or generality—forinstance, a “1-d” curve located anywhere on an image is just a sequenceof pixels, and hence a type of blob. Thus, both can use the samerepresentation, and hence all the same support functions (memorymanagement, etc).

Once a representation is chosen, any blob ‘extraction’ might be a singletwo-step operation. First: define the blob ‘body,’ second: copy pixelvalues from the image to their corresponding blob locations. (This canbe a ‘filter’ operation, and may follow any sequence of filter ops thatresulted in an image, as well as being applicable to a static image.)

Even for images, an “auction” process for processing can involve havingoperations available to convert from the internal format to and from theappropriate external one. For blobs and other features, quite a varietyof format conversions might be supported.

It's perhaps useful to digress a bit from a “normal” discussion of animage processing or computer vision package, to return to the nature ofapplications that may be run in the detailed arrangements, and the(atypical) constraints and freedoms involved.

For example, while some tasks will be ‘triggered’ by a direct useraction, others may simply be started, and expected to triggerthemselves, when appropriate. That is, a user might aim a smart phone ata parking lot and trigger a ‘find my car’ application, which would snapan image, and try to analyze it. More likely, the user would prefer totrigger the app, and then wander through the lot, panning the cameraabout, until the device signals that the car has been identified. Thedisplay may then present an image captured from the user's currentlocation, with the car highlighted.

While such an application may or may not become popular, it is likelythat many would contain processing loops in which images are acquired,sampled and examined for likely presence of a target, whose detectionwould trigger the ‘real’ application, which would bring morecomputational power to bear on the candidate image. The process wouldcontinue until the app and user agree that it has been successful, orapparent lack of success causes the user to terminate it. Desirably, the‘tentative detection’ loop should be able to run on the camera alone,with any outside resources called in only when there was reason to hopethat they might be useful.

Another type of application would be for tracking an object. Here, anobject of known type having been located (no matter how), a successionof images is thereafter acquired, and the new location of that objectdetermined and indicated, until the application is terminated, or theobject is lost. In this case, one might use external resources to locatethe object initially, and very likely would use them to specialize aknown detection pattern to the specific instance that had been detected,while the ensuing ‘tracking’ app, using the new pattern instance,desirably runs on the phone, unaided. (Perhaps such an application wouldbe an aid in minding a child at a playground.)

For some applications, the pattern recognition task may be prettycrude—keeping track of a patch of blue (e.g., a sweater) in a sequenceof frames, perhaps—while in others it might be highly sophisticated:e.g., authenticating a banknote. It is likely that a fairly small numberof control loops, like the two mentioned above, would be adequate for agreat many simple applications. They would differ in the featuresextracted, the pattern-matching technique employed, and the nature ofexternal resources (if any) resorted to.

As indicated, at least a few pattern recognition applications may runnatively on the basic mobile device. Not all pattern recognition methodswould be appropriate for such limited platforms. Possibilities wouldinclude: simple template matching, especially with a very smalltemplate, or a composite template using very small elements; Hough-stylematching, with modest resolution requirements for the detectedparameters; and neural-net detection. Note that training the net wouldprobably require outside resources, but applying it can be done locally,especially if a DSP or graphics chip can be employed. Any detectiontechnique that employs a large data-base lookup, or is toocomputationally intensive (e.g., N-space nearest-neighbor) is probablybest done using external resources.

More on Clumping

As noted earlier, clumping refers to a process for identifying groups ofpixels as being related.

One particular approach is to group scene items with a “common fate,”e.g., sharing common motion. Another approach relies on amulti-threshold or scale space tree. A data structure (including theblackboard) can store symbolic tags indicating the method(s) by which aclump was identified, or the clump can be stored with a label indicatingits type. (Recognition agents can contribute to a tag/label dictionary.)

The tags can derive from the clustering method used, and the featuresinvolved (e.g., color uniformity combined with brightness edges). At alowest level, “locally bright edges” or “most uniform color” may beused. At higher levels, tags such as “similar uniformity levels, nearbut separated by locally bright edges” can be employed. At still higherlevels, tags such as “like foliage” or “like faces” may be assigned toclumps—based on information from recognition agents. The result is ann-dimensional space populated with tagged features, facilitatinghigher-order recognition techniques (possibly as projections of featuresagainst specific planes).

Common motion methods consider 2D motions of points/features betweenimages. The motions can be, e.g., nearly identical displacement, ornearly linear displacement along an image direction, or nearly commonrotation around an image point. Other approaches can also be used, suchas optic flow, swarm of points, motion vectors, etc.

Multi-threshold tree methods can be used to associate a tree of nestedblobs within an image. FIGS. 20A and 20B are illustrative. Briefly, theimage (or an excerpt) is thresholded—with each pixel value examined todetermine whether it meets or exceeds a threshold. Initially thethreshold may be set to black. Every pixel passes this criterion. Thethreshold value is then raised. Parts of the image begin not to meet thethreshold test. Areas (blobs) appear where the threshold test is met.Eventually the threshold reaches a bright (high) level. Only a few smalllocations remain that pass this test.

As shown by FIGS. 20A and 20B, the entire image passes the blackthreshold. At a dark threshold, a single blob (rectangular) meets thetest. As the threshold is increased, two oval blob areas differentiate.Continuing to raise the threshold to a bright value causes the firstarea to separate into two bright ovals, and the second area to resolvedown to a single small bright area.

Testing the pixel values against such a varying threshold provides aquick and check way to identify related clumps of pixels within theimage frame.

In practical implementation, the image may first be processed with aGaussian or other blur to prevent slight noise artifacts from undulyinfluencing the results.

(Variants of this method can serve as edge detectors. E.g., if a contourof one of the blobs stays generally fixed while the threshold is raiseover several values, the contour is discerned to be an edge. Thestrength of the edge is indicated by the range of threshold values overwhich the contour is essentially fixed.)

While thresholding against luminance value was detailed, other thresholdmetrics can similarly be compared against, e.g., color, degree oftexture, etc.

Clumps identified by such methods can serve as organizing constructs forother data, such as image features and keyvectors. For example, oneapproach for identifying that features/keyvectors extracted from imagedata are related is to identify the smallest thresholded blob thatcontains them. The smaller the blob, the more related the featuresprobably are. Similarly, if first and second features are known to berelated, then other features that relate can be estimated by finding thesmallest thresholded blob that contains the first two features. Anyother features within that blob are also probably related to the firstand second features.

Freedoms and Constraints

Practicality of some pattern recognition methods is dependent on theplatform's ability to perform floating point operations, or invoke aDSP's vector operations, at an application's request.

More generally, there are a number of specific freedoms and constraintson an Intuitive Computing Platform. Freedoms include the ability oftasks to make use of off-device resources, whether on a nearbycommunicating accessory device or in the cloud, allowing applicationswhich “couldn't possibly” run on the device, seem to do so. Constraintsinclude: limited CPU power, limited available memory, and the need forapplications to proceed with varying resources. For instance, the memoryavailable might not only be limited, but might suddenly be reduced(e.g., a phone call is begun) and then made available again as thehigher priority application terminates.

Speed is also a constraint—generally in tension with memory. The desirefor a prompt response might push even mundane applications up against amemory ceiling.

In terms of feature representations, memory limits may encouragemaintaining ordered lists of elements (memory requirement proportionalto number of entries), rather than an explicit array of values (memoryrequirement proportional to the number of possible parameters).Operation sequences might use minimal buffers (as noted above) ratherthan full intermediate images. A long sequence of images might be“faked” by a short actual sequence along with one or more averagedresults.

Some “standard” imaging features, such as Canny edge operators, may betoo resource-intensive for common use. However, the same was formerlysaid about FFT processing—an operation that smart phone appsincreasingly employ.

On-Device Processing Suitable for Consideration

Within the context of the constraints above, the following outlinedetails classes of widely useful operations that may be included in therepertoire of the local device:

I. Task-related operations

-   -   A. Image related        -   i. Image sequence operations            -   a) extracting an image from the sequence            -   b) generating an image from a sequence range            -   c) tracking a feature or ROI through a sequence        -   ii. Image transformation            -   a) pointwise remapping            -   b) affine transformation            -   c) local operation: e.g., edge, local average, . . .            -   d) FFT, or related        -   iii. Visual feature extraction from image            -   a) 2D features            -   b) 1D features            -   c) 3D-ish features            -   d) full image->list of ROI            -   e) nonlocal features (color histogram, . . . )            -   f) scale, rotation-invariant intensity features        -   iv. feature manipulation            -   a) 2D features from 2D features            -   b) 1D to 1D etc            -   c) 1D features from 2D features        -   v. UI—image feedback (e.g., overlaying tag-related symbols            on image)    -   B. Pattern recognition        -   i. Extracting a pattern from a set of feature sets        -   ii. associating sequences, images, or feature sets with tags        -   iii. ‘recognizing’ a tag or tag set from a feature set        -   iv. ‘recognizing’ a composite or complex tag from a simpler            set of ‘recognized’ tags    -   C. App-related communication        -   i. Extract a list of necessary functions from a system state        -   ii. Broadcast a request for bids—collect responses        -   iii. transmit distilled data, receive outsources results

II. Action related operations (many will already be present among basicsystem actions)

-   -   i. activate/deactivate a system function    -   ii. produce/consume a system message    -   iii. detect the system state    -   iv. transition system to a new state    -   v. maintain queues of pending, active, and completed actions

User Experience and User Interface

One particular embodiment of the present technology allows an untraineduser to discover information about his environment (and/or about objectsin his presence) through use of a mobile device, without having todecide which tools to use, and while providing the ability to continuean interrupted discovery experience whenever and wherever desired.

The reader will recognize that existing systems, such as the iPhone, donot meet such needs. For example, the user must decide which one(s) ofthousands of different iPhone applications should be launched to provideinformation of the particular type desired. And if the user isinterrupted while directing the operation, there is no way of resumingthe discovery process at a later time or place. That is, the user mustexperience the discovery at the point of interaction with the object orenvironment. There is no ability to “save” the experience for laterexploration or sharing.

FIG. 19 shows a smart phone 100 with an illustrative user interfaceincluding a screen 102 and a discover button 103.

The discover button 103 is hardwired or programmed to cause the phone toactivate its discovery mode—analyzing incoming stimuli to discernmeaning and/or information. (In some modalities the phone is alwaysanalyzing such stimulus, and no button action is needed.)

Depicted screen 102 has a top pane portion 104 and a lower pane portion106. The relative sizes of the two panes is controlled by a bar 108,which separates the depicted panes. The bar 108 can be dragged by theuser to make the top pane larger, or the bottom pane larger, usingconstructs that are familiar to the graphical user interface designer.

The illustrative bottom pane 106 serves to present spatial information,such as maps, imagery, GIS layers, etc. This may be termed a geolocationpane, although this should not be construed as limiting itsfunctionality.

The illustrative top pane 104 is termed the sensor pane in the followingdiscussion—although this again is not limiting. In the mode shown, thispane presents audio information, namely an auditory scene visualization.However, a button 131 is presented on the UI by which this top pane canbe switched to present visual information (in which case button thenreads AUDIO—allowing the user to switch back). Other types of sensordata, such as magnetometer, accelerometer, gyroscope, etc., can bepresented in this pane also.

Starting with the top pane, one or more audio sensors (microphones) inthe smart phone listens to the audio environment. Speaker/speechrecognition software analyzes the captured audio, to attempt to identifyperson(s) speaking, and discern the words being spoken. If a match ismade (using, e.g., stored speaker characterization data stored locallyor in the cloud), an icon 110 corresponding to the identified speaker ispresented along an edge of the display. If the smart phone has access toa stored image 110 a of a recognized speaker (e.g., from the user'sphonebook or from Facebook), it can be used as the icon. If not, adefault icon 110 b can be employed. (Different default icons may beemployed for male and female speakers, if the recognition software canmake a gender determination with a specified confidence threshold.) Theillustrated UI shows that two speakers have been detected, although inother situations there may be more or fewer.

In addition to speech recognition, processes such as watermark detectionand fingerprint calculation/lookup can be applied to the audio streamsto identify same. By these or other approaches the software may detectmusic in the ambient audio, and present an icon 112 indicating suchdetection.

Other distinct audio types may also be detected and indicated (e.g.,road noise, birdsongs, television, etc., etc.)

To the left of each of the icons (110, 112, etc.) is a waveform display120. In the depicted embodiment, waveforms based on actual data aredisplayed, although canned depictions can be used if desired. (Otherforms of representation can be used, such as spectral histograms.) Theillustrated analog waveforms move to the left, with the newest data tothe right (akin to our experience in reading a line of text). Only themost recent interval of each waveform is presented (e.g., 3, 10 or 60seconds) before moving out of sight to the left.

The segmentation of the ambient audio into distinct waveforms is anapproximation; accurate separation is difficult. In a simple embodimentemploying two different microphones, a difference signal between the twoaudio streams is determined—providing a third audio stream. When thefirst speaker is sensed to be speaking, the stronger of these threesignals is presented (waveform 120 a). When that speaker is notspeaking, that waveform (or another) is presented at a greatlyattenuated scale—indicating that he has fallen silent (although theambient audio level may not have diminished much in level).

Likewise with the second speaker, indicated by icon 110 b. When thatperson's voice is recognized (or a human voice is discerned, but notidentified—but known not be to be the speaker indicated by icon 110 a),then the louder of the three audio signals is displayed in waveform form120 b. When that speaker falls silent, a much-attenuated waveform ispresented.

A waveform 120 c is similarly presented to indicate the sensedbackground music. Data from whichever of the three sources is leastcorrelated with the speakers' audio may be presented. Again, if themusic is interrupted, the waveform can be attenuated by the software toindicate same.

As noted, only a few seconds of audio is represented by the waveforms120. Meanwhile, the smart phone is analyzing the audio, discerningmeaning. This meaning can include, e.g., speech recognition text for thespeakers, and song identification for the music.

When information about an audio stream is discerned, it can berepresented by a bauble (icon) 122. If the bauble corresponds to anexcerpt of audio that is represented by a waveform still traversing thescreen, the bauble can be placed adjacent the waveform, such as bauble122 a (which can indicate, e.g., a text file for the speaker's recentutterance). The bauble 122 a moves with the waveform to which itcorresponds, to the left, until the waveform disappears out of sight ata virtual stop-gate 123. At that point the bauble is threaded onto ashort thread 124.

Baubles 122 queue up on thread 124, like pearls on a string. Thread 124is only long enough to hold a limited number of baubles (e.g., two tofive). After the thread is full, each added bauble pushes the oldest outof sight. (The disappearing bauble is still available in the history.)If no new baubles arrive, existing baubles may be set to “age-out” afteran interval of time, so that they disappear from the screen. Theinterval may be user-configured; exemplary intervals may be 10 or 60seconds, or 10 or 60 minutes, etc.

(In some embodiments, proto-baubles may be presented in association withwaveforms or other features even before any related information has beendiscerned. In such case, tapping the proto-bauble causes the phone tofocus its processing attention on obtaining information relating to theassociated feature.)

The baubles 122 may include visible indicia to graphically indicatetheir contents. If, for example, a song is recognized, the correspondingbauble can contain associated CD cover artwork, the face of the artist,or the logo of the music distributor (such as baubles 122 b).

Another audio scene visualization identifies, and depicts, differentaudio streams by reference to their direction relative to the phone. Forexample, one waveform might be shown as incoming from the upper right;another may be shown as arriving from the left. A hub at the centerserves as the stop-gate for such waveforms, against which baubles 122accumulate (as on strings 124). Tapping the hub recalls the storedhistory information. Such an arrangement is shown in FIG. 19A.

A history of all actions and discoveries by the smart phone may becompiled and stored—locally and/or remotely. The stored information caninclude just the discovered information (e.g., song titles, spoken text,product information, TV show titles), or it can include more—such asrecordings of the audio streams, and image data captured by the camera.If the user elects by appropriate profile settings, the history caninclude all data processed by the phone in session, includingkeyvectors, accelerometer and all other sensor data, etc.

In addition, or alternatively, the user interface can include a “SAVE”button 130. User activation of this control causes the information stateof the system to be stored. Another user control (not shown) allows thestored information to be restored to the system, so device analysis anduser discovery can continue—even at a different place and time. Forexample, if a user is browsing books at a bookstore, and a pager summonshim to an available table at a nearby restaurant, the user can pressSAVE. Later, the session can be recalled, and the user can continue thediscovery, e.g., with the device looking up a book of interest byreference to its jacket art or barcode, and with the device identifyinga song that was playing in the background.

While FIG. 19 shows information about the audio environment in thesensor pane 104, similar constructs can be employed to presentinformation about the visual environment, e.g., using arrangementsdetailed elsewhere in this specification. As noted, tapping the CAMERAbutton 131 switches modalities from audio to visual (and back). In thevisual mode this sensor pane 104 can be used to display augmentedreality modes of interaction.

Turning to the lower, geolocation pane 106 of FIG. 19, map data isshown. The map may be downloaded from an online service such as GoogleMaps, Bing, etc.

The resolution/granularity of the map data initially depends on thegranularity with which the smart phone knows its present location. Ifhighly accurate location information is known, a finely detailed map maybe presented (e.g., zoomed-in); if only gross location is known, a lessdetailed map is shown. The user may zoom in or out, to obtain more orless detail, by a scale control 140, as is conventional. The user'slocation is denoted by a larger push pin 142 or other indicia.

Each time the user engages in a discovery session, or a discoveryoperation, e.g., by tapping a displayed bauble, a smaller pin 146 islodged on the map—memorializing the place of the encounter. Informationabout the discovery operation (including time and place) is stored inassociation with the pin.

If the user taps a pin 146, information about the prior discovery isrecalled from storage and presented in a new window. For example, if theuser had a discovery experience with a pair of boots at the mall, animage of the boots may be displayed (either user-captured, or a stockphoto), together with price and other information presented to the userduring the earlier encounter. Another discovery may have involvedrecognition of a song at a nightclub, or recognition of a face in aclassroom. All such events are memorialized by pins on the displayedmap.

The geolocation pane facilitates review of prior discoveries, by a timecontrol 144 (e.g., a graphical slider). At one extreme, no previousdiscoveries are indicated (or only discoveries within the past hour).However, by varying the control, the map is populated with additionalpins 146—each indicating a previous discovery experience, and thelocation at which it took place. The control 144 may be set to show,e.g., discoveries within the past week, month or year. A “H” (history)button 148 may be activated to cause slider 144 to appear—allowingaccess to historical discoveries.

In some geographical locations (e.g., a mall, or school), the user'shistory of discoveries may be so rich that the pins must be filtered soas not to clutter the map. Thus, one mode allows start- and end-date ofdiscoveries to be user-set (e.g., by a pair of controls like slider144). Or keyword filters may be applied through a corresponding UIcontrol, e.g., Nordstrom, boot, music, face, peoples' names, etc.

A compass arrow 146 is presented on the display, to aid in understandingthe map. In the depicted mode, “up” on the map is the direction towardswhich the phone is oriented. If the arrow 146 is tapped, the arrow snapsto a vertical orientation. The map is then rotated so that “up” on themap corresponds to north.

The user can make available for sharing with others as much or as littleinformation about the user's actions as desired. In one scenario, auser's profile allows sharing of her discoveries at the local mall, butonly with selected friends on her FaceBook social network account, andonly if the user has expressly saved the discovery (as opposed to thesystem's history archive, which normally logs all actions). If shediscovers information about a particular book at the bookstore, andsaves the discovery, this information is posted to a data store cloud.If she returns to the mall a week later, and reviews baubles fromearlier visits, she may find that a friend was at the bookstore in themeantime and looked at the book, based on the user's stored discoveryexperience. That friend may have posted comments about the book, andpossibly recommended another book on the same subject. Thus, cloudarchives about discoveries can be shared for others to discover andaugment with content of their own.

Similarly, the user may consent to make some or all of the user'sdiscovery history available to commercial entities, e.g., for purposessuch as audience measurement, crowd traffic analysis, etc.

Illustrative Sequences of Operations

It will be understood that the FIG. 19 arrangement can be presented withno user interaction. The displayed mode of operation can be the device'sdefault, such as a screen saver to which the device reverts followingany period of inactivity.

In one particular arrangement, the software is activated when the phoneis picked up. The activation can be triggered by device movement orother sensor event (e.g., visual stimulus change, or sensing a tap onthe screen). In the first second or so of operation, the camera andmicrophone are activated, if not already. The phone makes a quickapproximation of position (e.g., by identifying a local WiFi node, orother gross check), and available location information is written to theblackboard for other processes to use. As soon as some locationinformation is available, corresponding map data is presented on thescreen (a cached frame of map data may suffice, if the phone's distancefrom the location to which the center of the map corresponds does notexceed a stored threshold, such as 100 yards, or a mile). The phone alsoestablishes a connection to a cloud service, and transmits the phone'slocation. The user's profile information is recalled, optionallytogether with recent history data.

Between one and three seconds of activation, the device starts toprocess data about the environment. Image and/or audio scenesegmentation is launched. Features noted in captured imagery may bedenoted by a proto-bauble displayed on the screen (e.g., here's a brightarea in the imagery that might be notable; this, over here, might beworth watching too . . . ). Keyvectors relating to sensed data can startstreaming to a cloud process. A more refined geolocation can bedetermined, and updated map data can be obtained/presented. Push pinscorresponding to previous discovery experiences can be plotted on themap. Other graphical overlays may also be presented, such as iconsshowing the location of the users' friends. If the user is downtown orat a mall, another overlay may show stores, or locations within stores,that are offering merchandise on sale. (This overlay may be provided onan opt-in basis, e.g., to members of a retailer's frequent shopper club.RSS-type distribution may feed such subscription information to thephone for overlay presentation.) Another overlay may show currenttraffic conditions on nearby roadways, etc.

Conspicuous features of interest may already be identified within thevisual scene (e.g., barcodes) and highlighted or outlined in a cameraview. Results of fast image segmentation operations (e.g., that's aface) can be similarly noted, e.g., by outlining rectangles. Results ofdevice-side recognition operations may appear, e.g., as baubles on thesensor pane 104. The bauble UI is activated, in the sense that it can betapped, and will present related information. Baubles can similarly bedragged across the screen to signal desired operations.

Still, the user has taken no action with the phone (except, e.g., tolift it from a pocket or purse).

If the phone is in the visual discovery mode, object recognition datamay start appearing on the sensor pane (e.g., locally, or from thecloud). It may recognize a box of Tide detergent, for example, andoverlay a correspondingly-branded bauble.

The user may drag the Tide bauble to different corners of the screen, tosignal different actions. One corner may have a garbage pail icon.Another corner may have a SAVE icon. Dragging it there adds it to ahistory data store that may be later recalled and reviewed to continuethe discovery.

If the user taps the Tide bauble, any other baubles may be greyed-out onthe screen. The phone shunts resources to further analysis of the objectindicated by the selected bauble—understanding the tap to be a userexpression of interest/intent.

Tapping the bauble can also summon a contextual menu for that bauble.Such menus can be locally-sourced, or provided from the cloud. For Tide,the menu options may include use instructions, a blog by which the usercan provide feedback to the manufacturer, etc.

One of the menu options can signal that the user wants further menuoptions. Tapping this option directs the phone to obtain other, lesspopular, options and present same to the user.

Alternatively, or additionally, one of the menu options can signal thatthe user is not satisfied with the object recognition results. Tappingthis option directs the phone (and/or cloud) to churn more, to try andmake a further discovery.

For example, a user in a bookstore may capture an image of a book jacketthat depicts Albert Einstein. The phone may recognize the book, andprovide links such as book reviews and purchasing options. The user'sintent, however, may have been to obtain further information aboutEinstein. Telling the phone to go back and work some more may lead tothe phone recognizing Einstein's face, and then presenting a set oflinks relating to the person rather, than the book.

In some user interfaces the menu options may have alternate meanings,depending on whether they are tapped once, or twice. A single tap on aparticular menu option may indicate that the user wants more menuoptions displayed. Two taps on the same menu option may signal that theuser is not satisfied with the original object recognition results, andwants others. The dual meanings may be textually indicated in thedisplayed menu legend.

Alternatively, conventions may arise by which users can infer the menumeaning of two taps, given the meaning of a single tap. For example, asingle tap may indicate instruction to perform an indicated task usingthe phone's local resources, whereas a double-tap directs performance ofthat same task by cloud resources. Or a single tap may indicateinstruction to perform the indicated task using computer resourcesexclusively, whereas a double-tap may indicate instruction to refer thetask for human-aided performance, such as by using Amazon's MechanicalTurk service.

Instead of tapping a bauble, a user may indicate interest by circlingone or more baubles—tracing a finger around the graphic on the screen.This form of input allows a user to indicate interest in a group ofbaubles.

Such a gesture (indicating interest in two or more baubles) can be usedto trigger action different than simply tapping two baubles separately.For example, circling the Apple and NASA baubles in FIG. 24 within acommon circle can direct the system to seek information that relates toboth Apple and NASA. In response, the device may provide information,e.g., on the NASA iPhone ap, which makes NASA imagery available to usersof the iPhone. Such discovery would not have arisen by tapping the Appleand NASA logos separately. Similarly, circling the NASA logo and theRolling Stones logo, together, may trigger a search leading to discoveryof a Wikipedia article about inclusion of a Rolling Stones song on agold-plated copper disk included aboard the Voyager spacecraft (afiction—introduced by the movie Starman).

FIG. 21A shows a discovery UI somewhat different from FIG. 19. Visualdiscovery occupies most of the screen, with the bottom band of thescreen displaying sensed audio information. Although not conspicuous inthis black and white depiction, across the center of the FIG. 21A screenis an overlayed red bauble 202 consisting of a stylized letter “O”(using the typeface from the banner of the Oregonian newspaper). In thiscase, the phone sensed a digital watermark signal from an article in theOregonian—triggering display of the bauble.

Clicking on the bauble causes it to transform, in animated fashion, intothe context-sensitive menu shown in FIG. 21B. At the center is a graphicrepresenting the object discovered in FIG. 21A (e.g., an article in thenewspaper). At the upper left is a menu item by which the user can mailthe article, or a link, to others. At the upper right is a menu itempermitting the article to be saved in a user archive.

At the lower left is a link to a blog on which the user can writecommentary relating to the article. At the lower right is a link to avideo associated with the article.

A reader of the newspaper may next encounter an advertisement for acasino. When sensed by the phone, a bauble again appears. Tapping thebauble brings up a different set of menu options, e.g., to buy ticketsto a performer's upcoming concert, to enter a contest, and to take a 360degree immersive tour of the casino hall. A “save” option is alsoprovided. At the center of the screen is a rectangle with the casino'slogo.

Viewing a digitally watermarked pharmaceutical bottle brings up yetanother context menu, shown in FIG. 22. At the center is an image ofwhat the pills should look like—allowing a safety check when takingmedicines (e.g., from a bottle in which a traveler has co-mingledseveral different pills). The medicine is also identified by name(“Fedratryl”), strength (“50 mg”) and by the prescribing doctor (“LeslieKatz”). One menu option causes the phone to call the user's doctor (orpharmacist). This option searches the user's phone book for theprescribing doctor's name, and dials that number. Another option submitsan automated prescription refill request to the pharmacy. Another linkleads to a web site presenting frequently asked questions about thedrug, and including FDA-required disclosure information. Another mayshow a map centered on the user's present locations—with push pinsmarking pharmacies that stock Fedratryl. Holding the phone vertically,rather than flat, switches the view to a markerless augmented realitypresentation, showing logos of pharmacies stocking Fedratryl thatappear, and disappear, overlaid on imagery of the actual horizon as thephone is moved to face different directions. (The 3DAR augmented realitySDK software for the iPhone, from SpotMetrix of Portland, Oreg., is usedfor the augmented reality presentation in an illustrative embodiment.) A“save” option is also provided.

In like fashion, a watermark in a PDF document can revealdocument-specific menu options; a barcode on a Gap jeans tag can lead tocare instructions and fashion tips; recognition of artwork on a bookjacket can trigger display of menu options including book reviews andpurchase opportunities; and recognition of a face can bring up optionssuch as viewing the person's FaceBook page, storing the name-annotatedphoto on Rich, etc. Similarly, watermarked radio or televisionaudio/video can lead to discovery of information about the sampledprogram, etc.

In some arrangements, digital signage (e.g., in a retail store) canpresent visual (or audio) content that is steganographically encodedwith watermark data. For example, a store may show a video presentationadvertising certain jeans. The video can be encoded with a plural bitpayload, e.g., conveying index data that can be used to access relatedinformation in a corresponding database record at a remote server. Thisrelated information can include, among other information, geolocationcoordinate data identifying the location of the signage from which thevideo watermark was decoded. This information can be returned to theuser's device, and used to inform the device of its location. In somecases (e.g., if the device is indoors), other location data—such as fromGPS satellites—may be unavailable. Yet the data returned from the remoteserver—corresponding to the decoded watermark information—providesinformation by which the phone can obtain or provide otherlocation-based services (even those unrelated to the store, thewatermark, etc.). For example, knowing that the device is atgeocoordinates corresponding, e.g., to a particular shopping mall, thephone may offer coupons or other information related to nearby merchants(e.g., by the same software application, by another, or otherwise).

FIG. 23 depicts a “radar” user interface clue associated with imageprocessing. An illuminated red bar 202 (shown in FIG. 24A) sweepsrepeatedly across the image—from a virtual pivot point. (This pivotpoint is off-screen, in the depicted cases.) The sweep alerts the userto the phone's image processing activity. Each sweep can indicate a newanalysis of the captured data.

Digital watermarks typically have an orientation that must be discernedbefore the watermark payload can be detected. Detection is facilitatedif the captured image is oriented in general alignment with thewatermark's orientation. Some watermarks have an orientation signal thatcan be quickly discerned to identify the watermark's orientation.

In the screen shot of FIG. 23B, the radar trace 202 causes a momentaryghost pattern to appear in its wake. This pattern shows a grid alignedwith the watermark orientation. Seeing an inclined grid (such asdepicted in FIG. 23B) may prompt the user to re-orient the phoneslightly, so that the grid lines are parallel to the screen edges—aidingwatermarking decoding.

As another visual clue—this one temporal, baubles may lose their spatialmoorings and drift to an edge of the screen after a certain time haselapsed. Eventually they may slip out of sight (but still be availablein the user's history file). Such an arrangement is shown in FIG. 24.(In other embodiments, the baubles stay spatially associated with imagefeatures—disappearing only when the associated visual features move outof view. For audio, and optionally for imagery, baubles mayalternatively effervesce in place with the passage of time.)

Audio discovery can parallel the processes detailed above. Proto-baublescan be immediately associated with detected sounds, and refined intofull baubles when more information is available. Different types ofaudio watermark decoding and fingerprinting/lookups can be used toidentify songs, etc. Speech recognition can be on-going. Some audio maybe quickly processed locally, and undergo more exhaustive processing inthe cloud. A bauble resulting from the local processing may take on adifferent appearance (e.g., bolded, or brighter, or in color vs.monochrome) once cloud processing is completed and confirms the originalconclusion. (Likewise for visual analysis, when a first identificationis confirmed—either by local and cloud processing, or by alternateidentification mechanisms, e.g., SIFT and barcode reading.)

As before, the user can tap baubles to reveal associated information andcontextual menus. When one bauble is tapped, processing of other objectsis suspended or reduced, so that processing can focus where the user hasindicated interest. If the user taps one of the displayed menu options,the device UI changes to one that supports the selected operation.

For a recognized song, the contextual menu may include a center panepresenting the artist name, track name, distributor, CD name, CDartwork, etc. Around the periphery can be links, e g, allowing the userto purchase the music at iTunes or Amazon, or see a YouTube music videoof the song. For spoken audio, a tap may open a menu that displays atranscript of the speaker's words, and offering options such as sendingto friends, posting to FaceBook, playing a stored recording of thespeaker's speech, etc.

Due to the temporal nature of audio, the user interface desirablyincludes a control allowing user access to information from an earliertime—for which baubles may have already been removed from the screen.One approach is to allow the user to sweep a desired audio trackbackwards (e.g., waveform 120 b to the right). This action suspendsongoing display of the waveform (although all the information isbuffered), and instead sequentially recalls audio, and associatedbaubles, from the stored history. When a desired bauble is restored tothe screen in such fashion, the user can tap it for the correspondingdiscovery experience. (Other devices for navigating the time domain canalternatively be provided, e.g., a shuttle control.)

To facilitate such temporal navigation, the interface may provide adisplay of relative time information, such as tic codes every 10 or 60seconds along the recalled waveform, or with textual timestampsassociated with recalled baubles (e.g., “2:45 ago”).

The software's user interface can include a “Later” button or the like,signaling that the user will not be reviewing discovery information inreal time. A user at a concert, for example, may activate thismode—acknowledging that her attention will be focused elsewhere.

This control indicates to the phone that it need not update the displaywith discovery data, nor even process the data immediately. Instead, thedevice can simply forward all of the data to the cloud for processing(not just captured audio and image data, but also GPS location,accelerometer and gyroscope information, etc.). Results from the cloudcan be stored in the user's history when done. At a later, moreconvenient time, the user may recall the stored data and explore thenoted discoveries—perhaps richer in their detail because they were notprocessed under the constraint of immediacy.

Another user interface feature can be a “dock” to which baubles aredragged and where they stick, e.g., for later access (akin to the dockin Apple's OS X operating system). When a bauble is docked in suchfashion, all keyvectors associated with that bauble are saved.(Alternatively, all keyvectors associated with the current session aresaved—providing more useful context for later operations.) Devicepreferences can be set so that if a bauble is dragged to the dock,related data (either bauble-specific, or the entire session) isprocessed by the cloud to discern more detailed information relating tothe indicated object.

Still another interface feature can be a “wormhole” (or SHARE icon) towhich baubles can be dragged. This posts the bauble, or relatedinformation (e.g., bauble-related keyvectors, or the entire sessiondata) for sharing with the user's friends. Baubles deposited into thewormhole can pop up on devices of the user's friends, e.g., as adistinctive pin on a map display. If the friend is accompanying theuser, the bauble may appear on the camera view of the friend's device,as an overlay on the corresponding part of the scene as viewed by thefriend's device. Other displays of related information can of course beused.

MAUI Project

Microsoft Research, at its TechFest 2010 event, publicized the MobileAssistance Using Infrastructure project, or MAUI.

An abstract of a paper by MAUI researcher Cuervo et al, MAUI: MakingSmartphones Last Longer With Code Offload, ACM MobiSys '10, introducesthe MAUI project as follows:

-   -   This paper presents MAUI, a system that enables fine-grained        energy-aware offload of mobile code to the infrastructure.        Previous approaches to these problems either relied heavily on        programmer support to partition an application, or they were        coarse-grained requiring full process (or full VM) migration.        MAUI uses the benefits of a managed code environment to offer        the best of both worlds: it supports fine-grained code offload        to maximize energy savings with minimal burden on the programmer        MAUI decides at run-time which methods should be remotely        executed, driven by an optimization engine that achieves the        best energy savings possible under the mobile device's current        connectivity constrains. In our evaluation, we show that MAUI        enables: 1) a resource-intensive face recognition application        that consumes an order of magnitude less energy, 2) a        latency-sensitive arcade game application that doubles its        refresh rate, and 3) a voice-based language translation        application that bypasses the limitations of the smartphone        environment by executing unsupported components remotely.

The principles and concepts noted by the MAUI researchers (includingindividuals from Duke, Carnegie Mellon, AT&T Research and LancasterUniversity) echo many of the principles and concepts in applicants'present and prior work. For example, their work is motivated by theobservation that battery constraints are a fundamental limitation on useof smart phones—an observation made repeatedly in applicants' work. Theypropose breaking cognition-related applications into sub-tasks, whichcan run either on a smartphone, or be referred to a cloud resource forexecution, as do applicants. They further propose that this allocationof different tasks to different processors can depend on dynamiccircumstances, such as battery life, connectivity, etc.—again echoingapplicants. The researchers also urge reliance on nearby processingcenters (“cloudlets”) for minimal latency—just as applicants proposedthe use of femtocell processing nodes on the edges of wireless networksfor this reason (application 61/226,195, filed Jul. 16, 2009; andpublished application WO2010022185).

In view of the many common aims and principles between the MAUI projectand the applicants' present and prior work, the reader is referred tothe MAUI work for features and details that can be incorporated into thepresent applicants' detailed arrangements. Similarly, features anddetails from the present applicants' work can be incorporated into thearrangements proposed by the MAUI researchers. By such integration,benefits accrue to each.

For example, MAUI employs the Microsoft .NET Common Language Runtime(CLR), by which code can be written once, and then run either on thelocal processor (e.g., an ARM CPU), or on a remote processor (typicallyan x86 CPU). In this arrangement, software developers annotate whichmethods of an application may be offloaded for remote execution. Atrun-time, a solver module analyzes whether each method should beexecuted remotely or locally, based on (1) energy consumptioncharacteristics, (2) program characteristics (e.g., running time andresource needs, and (3) network characteristics (e.g., bandwidth,latency and packet loss). In particular, the solver module constructsand solves a linear programming formulation of the code offload problem,to find an optimal partitioning strategy that minimizes energyconsumption, subject to latency constraints.

Similarly, the MAUI researchers detail particular cloudletarchitectures, and virtual machine synthesis techniques, than can beemployed advantageously in conjunction with applicants' work. They alsodetail transient customization methods that restore the cloudlet to itspristine software state after each use—encapsulating the transient guestsoftware environment from the permanent host software environment of thecloudlet infrastructure, and defining a stable ubiquitous interfacebetween the two. These and the other MAUI techniques can be directlyemployed in embodiments of applicants' technology.

Additional information on MAUI is found in a paper by Satyanarayanan etal, “The Case for VM-based Cloudlets in Mobile Computing,” IEEEPervasive Computing, Vol. 8, No. 4, pp 14-23, November, 2009 (attachedas Appendix A in incorporated-by-reference document 61/318,217, whichwill be available for public inspection upon the publication of thisapplication). Still further information is found in a write-up posted tothe web on Mar. 4, 2010, entitled “An Engaging Discussion” (attached asAppendix B to application 61/318,217). The artisan is presumed to befamiliar with such prior work.

More on Sound Source Localization

As smart phones become ubiquitous, they can cooperate in novel ways. Oneis to perform advanced sound source localization.

As is known from the prior art (e.g., US20080082326 and US20050117754),signals from spatially separated microphones can be used to discern thedirection from which audio emanates, based on time delays betweencorrelated features in the sensed audio signals. Phones carried bydifferent individuals can serve as the spatially separated microphones.

A prerequisite to sound source localization is understanding thepositions of the component audio sensors. GPS is one location technologythat can be used. However, more accurate technologies are emerging, someof which are noted below. Using such technologies, relative locations ofcell phones may be determined to within an accuracy of less than a meter(in some cases closer to a centimeter).

Such localization technologies can be used to identify the position ofeach cooperating phone in three spatial dimensions. Further refinementcan derive from knowing the location and orientation of the sensor(s) onthe phone body, and knowing the orientation of the phone. The formerinformation is specific to each phone, and may be obtained from local orremote data storage. Sensors in the phone, such as accelerometers,gyroscopes and magnetometers, can be used to provide the phoneorientation information. Ultimately, a 6D pose for each microphone maybe determined.

The phones then share this information with other phones. The phones canbe programmed to broadcast time-stamped digital streams of audio assensed by their microphones. (Data for several streams may be broadcastby a phone with several microphones.) Location information can also bebroadcast by each phone, or one phone may discern the location ofanother using suitable technology, as noted below. The broadcasts can beby short range radio technologies, such as Bluetooth or Zigbee or802.11. A service discovery protocol such as Bonjour can be used toexchange data between the phones, or another protocol can be used.

While MP3 compression is commonly used for audio compression, its use isnot favored in the present circumstance. MP3 and the like representaudio as serial sets of frequency coefficients, per a sampling window.This sampling window is, in effect, a window of temporal uncertainty.This uncertainty limits the accuracy with which a sound source can belocalized. In order for feature correlation to accurately be related totime delay, it is preferred that uncompressed audio, or compression thatfaithfully preserves temporal information (e.g., lossless datacompression) be used.

In one embodiment, a first phone receives audio data sensed by andbroadcast from one or more second phones, and—in conjunction with datasensed by its own microphone—judges the source direction of a sound.This determination may then be shared with other phones, so that they donot need to make their own determinations. The sound source location canbe expressed as a compass direction from the first phone. Desirably, thelocation of the first phone is known to the others, so that the soundsource localization information relative to the first phone can berelated to the positions of the other phones.

In another arrangement, a dedicated device within an environment servesto collect audio streams from nearby sensors, makes sound sourcelocalization determinations, and broadcasts its findings to theparticipating phones. This functionality may be built into otherinfrastructure devices, such as lighting controllers, thermostats, etc.

Determining audio direction in two dimensions is sufficient for mostapplications. However, if the microphones (phones) are spaced in threedimensions (e.g., at different elevations), then sound source directioncan be determined in three dimensions.

If the sensors are spaced by meters rather than centimeters (as iscommon in many applications, such as multiple microphones on a singlephone), the source of a sound can be localized not just by itsdirection, but also by its distance. Using triangulation based ondirectional information, and knowing their own respective locations, twoor more spatially-separated phones can determine the distance from eachto the sound source. Distance and direction from a known phone locationallows the position of the sound source to be determined. As before,this position information can be resolved in three dimensions, if thesensors are distributed in three dimensions. (Again, these calculationscan be performed by one phone, using data from the other. The resultinginformation can then be shared.)

Linked Data

In accordance with another aspect of the present technology, Web 2.0notions of data and resources (e.g., in connection with Linked Data) areused with tangible objects and/or related keyvector data, and associatedinformation.

Linked data refers to arrangements promoted by Sir Tim Berners Lee forexposing, sharing and connecting data via de-referenceable URIs on theweb. (See, e.g., T. B. Lee, Linked Data,www<dot>w3<dot>org/Designlssues/LinkedData.html.)

Briefly, URIs are used to identify tangible objects and associated dataobjects. HTTP URIs are used so that these objects can be referred to andlooked up (“de-refeerenced”) by people and user agents. When a tangibleobject is de-referenced, useful information (e.g., structured metadata)about the tangible object is provided. This useful information desirablyincludes links to other, related URIs—to improve discovery of otherrelated information and tangible objects.

RDF (Resource Description Framework) is commonly used to representinformation about resources. RDF describes a resource (e.g., tangibleobject) as a number of triples, composed of a subject, predicate andobject. These triples are sometimes termed assertions.

The subject of the triple is a URI identifying the described resource.The predicate indicates what kind of relation exists between the subjectand object. The predicate is typically a URI as well—drawn from astandardized vocabulary relating to a particular domain. The object canbe a literal value (e.g., a name or adjective), or it can be the URI ofanother resource that is somehow related to the subject.

Different knowledge representation languages can be used to expressontologies relating to tangible objects, and associated data. The WebOntology language (OWL) is one, and uses a semantic model that providescompatibility with the RDF schema. SPARQL is a query language for usewith RDF expressions—allowing a query to consist of triple patterns,together with conjunctions, disjunctions, and optional patterns.

According to this aspect of the present technology, items of datacaptured and produced by mobile devices are each assigned a unique andpersistent identifier. These data include elemental keyvectors,segmented shapes, recognized objects, information obtained about theseitems, etc. Each of these data is enrolled in a cloud-based registrysystem, which also supports related routing functions. (The dataobjects, themselves, may also be pushed to the cloud for long termstorage.) Related assertions concerning the data are provided to theregistry from the mobile device. Thus, each data object known to thelocal device is instantiated via data in the cloud.

A user may sweep a camera, capturing imagery. All objects (and relateddata) gathered, processed and/or identified through such action areassigned identifiers, and persist in the cloud. A day or a year later,another user can make assertions against such objects (e.g., that a treeis a white oak, etc.). Even a quick camera glance at a particular place,at a particular time, is memorialized indefinitely in the cloud. Suchcontent, in this elemental cloud-based form, can be an organizingconstruct for collaboration.

Naming of the data can be assigned by the cloud-based system. (The cloudbased system can report the assigned names back to the originatingmobile device.) Information identifying the data as known to the mobiledevice (e.g., clump ID, or UID, noted above) can be provided to thecloud-based registry, and can be memorialized in the cloud as anotherassertion about the data.

A partial view of data maintained by a cloud-based registry can include:

Subject Predicate Object TangibleObject#HouseID6789 Has_the_Color BlueTangibleObject#HouseID6789 Has_the_Geolocation 45.51N 122.67WTangibleObject#HouseID6789 Belongs_to_the_Neighborhood SellwoodTangibleObject#HouseID6789 Belongs_to_the_City PortlandTangibleObject#HouseID6789 Belongs_to_the_Zip_Code 97211TangibleObject#HouseID6789 Belongs_to_the_Owner Jane A. DoeTangibleObject#HouseID6789 Is_Physically_Adjacent_ToTangibleObject#HouseID6790 ImageData#94D6BDFA623Was_Provided_From_Device iPhone 3Gs DD69886 ImageData#94D6BDFA623Was_Captured_at_Time Nov. 30, 2009, 8:32:16 pm ImageData#94D6BDFA623Was_Captured_at_Place 45.51N 122.67W ImageData#94D6BDFA623Was_Captured_While_Facing 5.3 degrees E of N ImageData#94D6BDFA623Was_Produced_by_Algorithm Canny ImageData#94D6BDFA623Corresponds_to_Item Barcode ImageData#94D6BDFA623 Corresponds_to_ItemSoup can

Thus, in this aspect, the mobile device provides data allowing thecloud-based registry to instantiate plural software objects (e.g., RDFtriples) for each item of data the mobile device processes, and/or foreach physical object or feature found in its camera's field of view.Numerous assertions can be made about each (I am Canny data; I am basedon imagery captured at a certain place and time; I am a highly textured,blue object that is visible looking north from latitude X, longitude/Y,etc.).

Importantly, these attributes can be linked with data posted by otherdevices—allowing for the acquisition and discovery of new informationnot discernible by a user's device from available image data and contextalone.

For example, John's phone may recognize a shape as a building, but notbe able to discern its street address, or learn its tenants. Jane,however, may work in the building. Due to her particular context andhistory, information that her phone earlier provided to the registry inconnection with building-related image data may be richer in informationabout the building, including information about its address and sometenants. By similarities in geolocation information and shapeinformation, the building about which Jane's phone provided informationcan be identified as likely the same building about which John's phoneprovided information. (A new assertion can be added to the cloudregistry, expressly relating Jane's building assertions with John's, andvice-versa.) If John's phone has requested the registry to do so (and ifrelevant privacy safeguards permit), the registry can send to John'sphone the assertions about the building provided by Jane's phone. Theunderlying mechanism at work here may be regarded as mediatedcrowd-sourcing, wherein assertions are created within the policy andbusiness-rule framework that participants subscribe too.

Locations (e.g., determined by place, and optionally also by time) thathave a rich set of assertions associated with them provide for newdiscovery experiences. A mobile device can provide a simple assertion,such as GPS location and current time, as an entry point from which tostart a search or discovery experience within the linked data, or otherdata repository.

It should also be noted that access or navigation of assertions in thecloud can be influenced by sensors on the mobile device. For example,John may be permitted to link to Jane's assertions regarding thebuilding only if he is within a specific proximity of the building asdetermined by GPS or other sensors (e.g., 10 m, 30 m, 100 m, 300 m,etc.). This may be further limited to the case where John either needsto be stationary, or traveling at a walking pace as determined by GPS,accelerometers/gyroscopes or other sensors (e.g., less than 100 feet, or300 feet, per minute). Such restrictions based on data from sensors inthe mobile device can reduce unwanted or less relevant assertions (e.g.,spam, such as advertising), and provide some security against remote ordrive-by (or fly-by) mining of data (Various arrangements can beemployed to combat spoofing of GPS or other sensor data.)

Similarly, assertions stored in the cloud may be accessed (or newassertions about subjects may be made) only when the two involvedparties share some trait, such as proximity in geolocation, time, socialnetwork linkage, etc. (The latter can be demonstrated by reference to asocial network data store, such as Facebook or LinkedIn, showing thatJohn is socially linked to Jane, e.g., as friends.) Such use ofgeolocation and time parallels social conventions, i.e. when largegroups of people gather, spontaneous interaction that occurs can berewarding as there is a high likelihood that the members of the grouphave a common interest, trait, etc. Ability to access, and post,assertions, and the enablement of new discovery experiences based on thepresence of others follows this model.

Location is a frequent clue that sets of image data are related. Otherscan be used as well.

Consider an elephant researcher. Known elephants (e.g., in a preserve)are commonly named, and are identified by facial features (includingscars, wrinkles and tusks). The researcher's smart phone may submitfacial feature vectors for an elephant to a university database, whichexists to associate facial vectors with an elephant's name. However,when such facial vector information is submitted to the cloud-basedregistry, a greater wealth of information may be revealed, e.g., datesand locations of prior sightings, the names of other researchers whohave viewed the elephant, etc. Again, once correspondence between datasets is discerned, this fact can be memorialized by the addition offurther assertions to the registry.

It will be recognized that such cloud-based repositories of assertionsabout stimuli sensed by cameras, microphones and other sensors of mobiledevices may quickly comprise enormous stores of globally usefulinformation, especially when related with information in other linkeddata systems (a few of which are detailed at linkeddata<dot>org). Sincethe understanding expressed by the stored assertions reflects, in part,the profiles and histories of the individual users whose devicescontribute such information, the knowledge base is particularly rich.(Google's index of the web may look small by comparison.)

(In connection with identification of tangible objects, a potentiallyuseful vocabulary is the AKT (Advanced Knowledge Technologies) ontology.It has, as its top level, the class “Thing,” under which are twosub-classes: “Tangible-Thing” and “Intangible-Thing.” “Tangible-Thing”includes everything from software to sub-atomic particles, both real andimaginary (e.g., Mickey Mouse's car). “Tangible-Thing” has subclassesincluding “Location,” “Geographical-Region,” “Person,”“Transportation-Device,” and “Information-Bearing-Object.” Thisvocabulary can be extended to provide identification for objectsexpected to be encountered in connection with the present technology.)

Augmented Space

One application of the present technology is a function that presentsinformation on imagery (real or synthetic) concerning the night sky.

A user may point a smart phone at a particular point of the sky, andcapture an image. The image may not, itself, be used for presentationon-screen, due to the difficulties of capturing starlight in a smallhandheld imaging device. However, geolocation, magnetometer,accelerometer and/or gyroscope data can be sampled to indicate thelocation from, and orientation at which, the user pointed the camera.Night sky databases, such as the Google Sky project (available throughthe Google Earth interface), can be consulted to obtain datacorresponding to that portion of the key. The smart phone processor canthen reproduce this data on the screen, e.g., directly from the Googleservice. Or it can overlay icons, baubles, or other graphical indicia atlocations on the screen corresponding to the positions of stars in thepointed-to portion of the sky. Lines indicating the Greek (and/orIndian, Chinese, etc.) constellations can be drawn on the screen.

Although the stars themselves may not be visible in imagery captured bythe camera, other local features may be apparent (trees, houses, etc.).Star and constellation data (icons, lines, names) can be displayed atopthis actual imagery—showing where the stars are located relative to thevisible surroundings. Such an application may also include provision formoving the stars, etc., through their apparent arcs, e.g., with a slidercontrol allowing the user to change the displayed viewing time (to whichthe star positions correspond) forward and backward. The user may thusdiscover that the North Star will rise from behind a particular tree ata particular time this evening.

Other Comments

While this specification earlier noted its relation to the assignee'sprevious patent filings, and to the MAUI project, it bears repeating.These materials should be read in concert and construed together.Applicants intend that features in each disclosure be combined withfeatures in the others. Thus, for example, the arrangements and detailsdescribed in the present specification can be used in variantimplementations of the systems and methods described in application Ser.Nos. 12/271,772 and 12/490,980, and in the MAUI work, while thearrangements and details of the just-mentioned work can be used invariant implementations of the systems and methods described in thepresent specification. Similarly for the other noted documents. Thus, itshould be understood that the methods, elements and concepts disclosedin the present application be combined with the methods, elements andconcepts detailed in those cited documents. While some have beenparticularly detailed in the present specification, many have not—due tothe large number of permutations and combinations, and the need forconciseness. However, implementation of all such combinations isstraightforward to the artisan from the provided teachings.

Having described and illustrated the principles of our inventive workwith reference to illustrative features and examples, it will berecognized that the technology is not so limited.

For example, while reference has been made to mobile devices such assmart phones, it will be recognized that this technology finds utilitywith all manner of devices—both portable and fixed. PDAs, organizers,portable music players, desktop computers, laptop computers, tabletcomputers, netbooks, ultraportables, wearable computers, servers, etc.,can all make use of the principles detailed herein. Particularlycontemplated smart phones include the Apple iPhone, and smart phonesfollowing Google's Android specification (e.g., the G1 phone,manufactured for T-Mobile by HTC Corp., the Motorola Droid phone, andthe Google Nexus phone). The term “smart phone” (or “cell phone”) shouldbe construed to encompass all such devices, even those that are notstrictly-speaking cellular, nor telephones (e.g., the Apple iPaddevice).

(Details of the iPhone, including its touch interface, are provided inApple's published patent application 20080174570.)

Similarly, this technology also can be implemented using face-wornapparatus, such as augmented reality (AR) glasses. Such glasses includedisplay technology by which computer information can be viewed by theuser—either overlaid on the scene in front of the user, or blocking thatscene. Virtual reality goggles are an example of such apparatus.Exemplary technology is detailed in patent documents 7,397,607 and20050195128. Commercial offerings include the Vuzix iWear VR920, theNaturalpoint Trackir 5, and the ezVision X4 Video Glasses by ezGear. Anupcoming alternative is AR contact lenses. Such technology is detailed,e.g., in patent document 20090189830 and in Parviz, Augmented Reality ina Contact Lens, IEEE Spectrum, September, 2009. Some or all such devicesmay communicate, e.g., wirelessly, with other computing devices (carriedby the user or otherwise), or they can include self-contained processingcapability. Likewise, they may incorporate other features known fromexisting smart phones and patent documents, including electroniccompass, accelerometers, gyroscopes, camera(s), projector(s), GPS, etc.

Further out, features such as laser range finding (LIDAR) may becomestandard on phones (and related devices), and be employed in conjunctionwith the present technology. Likewise any other sensor technology, e.g.,tactile, olfactory, etc.

While the detailed technology made frequent reference to baubles, othergraphical icons—not necessarily serving the purpose of baubles in thedetailed arrangements, can be employed, e.g., in connection with userinterfaces.

The specification detailed various arrangements for limiting the baublesplaced on the user's screen, such as a verbosity control, scoringarrangements, etc. In some embodiments it is helpful to provide anon-programmable, fixed constraint (e.g., thirty baubles), so as toprevent a virus-based Denial of Service attack from overwhelming thescreen with baubles, to the point of rendering the interface useless.

While baubles as described in this specification are most generallyassociated with image and audio features, they can serve other purposesas well. For example, they can indicate to the user which tasks arepresently operating, and provide other status information.

It should be noted that commercial implementations of the presenttechnology will doubtless employ user interfaces wholly different thanthose presented in this specification. Those detailed in this documentare props to aid in explanation of associated technologies (although inmany instances their principles and features are believed to beinventive in their own rights). In like fashion, the detailed usermodalities of interaction are illustrative only; commercialimplementations will doubtless employ others.

The design of smart phones and other computer devices referenced in thisdisclosure is familiar to the artisan. In general terms, each includesone or more processors (e.g., of an Intel, AMD or ARM variety), one ormore memories (e.g. RAM), storage (e.g., a disk or flash memory), a userinterface (which may include, e.g., a keypad, a TFT LCD or OLED displayscreen, touch or other gesture sensors, a camera or other opticalsensor, a compass sensor, a 3D magnetometer, a 3-axis accelerometer,3-axis gyroscopes, a microphone, etc., together with softwareinstructions for providing a graphical user interface), interconnectionsbetween these elements (e.g., buses), and an interface for communicatingwith other devices (which may be wireless, such as GSM, CDMA, W-CDMA,CDMA2000, TDMA, EV-DO, HSDPA, WiFi, WiMax, mesh networks, Zigbee andother 802.15 arrangements, or Bluetooth, and/or wired, such as throughan Ethernet local area network, a T-1 internet connection, etc).

More generally, the processes and system components detailed in thisspecification may be implemented as instructions for computing devices,including general purpose processor instructions for a variety ofprogrammable processors, including microprocessors, graphics processingunits (GPUs, such as the nVidia Tegra APX 2600), digital signalprocessors (e.g., the Texas Instruments TMS320 series devices), etc.These instructions may be implemented as software, firmware, etc. Theseinstructions can also be implemented to various forms of processorcircuitry, including programmable logic devices, FPGAs (e.g., XilinxVirtex series devices), FPOAs (e.g., PicoChip brand devices), andapplication specific circuits—including digital, analog and mixedanalog/digital circuitry. Execution of the instructions can bedistributed among processors and/or made parallel across processorswithin a device or across a network of devices. Transformation ofcontent signal data may also be distributed among different processorand memory devices. References to “processors” or “modules” (such as aFourier transform processor, or an FFT module, etc.) should beunderstood to refer to functionality, rather than requiring a particularform of implementation.

Software instructions for implementing the detailed functionality can bereadily authored by artisans, from the descriptions provided herein,e.g., written in C, C++, Visual Basic, Java, Python, Tcl, Perl, Scheme,Ruby, etc. Mobile devices according to the present technology caninclude software modules for performing the different functions andacts. Known artificial intelligence systems and techniques can beemployed to make the inferences, conclusions, and other determinationsnoted above.

Commonly, each device includes operating system software that providesinterfaces to hardware resources and general purpose functions, and alsoincludes application software which can be selectively invoked toperform particular tasks desired by a user. Known browser software,communications software, and media processing software can be adaptedfor many of the uses detailed herein. Software and hardwareconfiguration data/instructions are commonly stored as instructions inone or more data structures conveyed by tangible media, such as magneticor optical discs, memory cards, ROM, etc., which may be accessed acrossa network. Some embodiments may be implemented as embedded systems—aspecial purpose computer system in which the operating system softwareand the application software is indistinguishable to the user (e.g., asis commonly the case in basic cell phones). The functionality detailedin this specification can be implemented in operating system software,application software and/or as embedded system software.

In addition to storing the software, the various memory componentsreferenced above can be used as data stores for the various informationutilized by the present technology (e.g., context information, tables,thresholds, etc.).

This technology can be implemented in various different environments.One is Android, an open source operating system available from Google,which runs on a Linux kernel. Android applications are commonly writtenin Java, and run in their own virtual machines Instead of structuringapplications as large, monolithic blocks of code, Android applicationsare typically implemented as collections of “activities” and “services,”which can be selectively loaded as needed. In one implementation of thepresent technology, only the most basic activities/services are loaded.Then, as needed, others are started. These can send messages to eachother, e.g., waking one another up. So if one activity looks forellipses, it can activate a face detector activity if a promisingellipse is located.

Android activities and services (and also Android's broadcast receivers)are activated by “intent objects” that convey messages (e.g., requestinga service, such as generating a particular type of keyvector). By thisconstruct, code can lie dormant until certain conditions arise. A facedetector may need an ellipse to start. It lies idle until an ellipse isfound, at which time it starts into action.

For sharing information between activities and services (e.g., servingin the role of the blackboard noted earlier), Android makes use of“content providers.” These serve to store and retrieve data, and make itaccessible to all applications.

Android SDKs, and associated documentation, are available atdeveloper<dot>android<dot>com/index.html.

Different of the functionality described in this specification can beimplemented on different devices. For example, in a system in which asmart phone communicates with a server at a remote service provider,different tasks can be performed exclusively by one device or the other,or execution can be distributed between the devices. Extraction ofbarcode, or eigenvalue, data from imagery are but two examples of suchtasks. Thus, it should be understood that description of an operation asbeing performed by a particular device (e.g., a smart phone) is notlimiting but exemplary; performance of the operation by another device(e.g., a remote server, or the cloud), or shared between devices, isalso expressly contemplated. (Moreover, more than two devices maycommonly be employed. E.g., a service provider may refer some tasks,such as image search, object segmentation, and/or image classification,to servers dedicated to such tasks.)

In like fashion, description of data being stored on a particular deviceis also exemplary; data can be stored anywhere: local device, remotedevice, in the cloud, distributed, etc.

Operations need not be performed exclusively byspecifically-identifiable hardware. Rather, some operations can bereferred out to other services (e.g., cloud computing), which attend totheir execution by still further, generally anonymous, systems. Suchdistributed systems can be large scale (e.g., involving computingresources around the globe), or local (e.g., as when a portable deviceidentifies nearby devices through Bluetooth communication, and involvesone or more of the nearby devices in a task—such as contributing datafrom a local geography; see in this regard patent 7,254,406 to Beros.)

Similarly, while certain functions have been detailed as being performedby certain modules, agents, processes, etc., in other implementationssuch functions can be performed by other of such entities, or otherwise(or dispensed with altogether).

Reference is sometimes made to “recognition agents,” and sometimes to“operations,” while other times to “functions,” and sometimes to“applications” or “services” or “modules” or “tasks” or “stages,” etc.In different software development environments these terms may havedifferent particular meanings. In the present specification, however,these terms are generally used interchangeably.

As noted, many functions can be implemented by a sequential operation ofplural component stages. Such functions may be regarded as multi-stage(cascaded) classifiers, in which the later stages only consider regionsor values that have been processed the earlier stages. For manyfunctions of this type, there can be a threshold or similar judgmentthat examines the output from one stage, and only activates the nextstage if a criterion is met. (The barcode decoder, which triggered onlyif a parameter output by a preceding stage had a value in excess of15,000, is one example of this type.)

In many embodiments, the functions performed by various components, aswell as their inputs and outputs, are specified or published (e.g., bythe components) in the form of standardized metadata, so that same canbe identified, such as by the dispatch process. The XML-based WSDLstandard can be used in some embodiments. (See, e.g., Web ServicesDescription Language (WSDL) Version 2.0 Part 1: Core Language, W3C,June, 2007.) An extension of WSDL, termed WSDL-S, extends WSDL toinclude semantic elements that improve reusability by, among otherfeatures, facilitating the composition of services. (An alternativesemantic-capable standard is the Ontology Web Language for Services:OWL-S.) For communicating with cloud-based service providers, theXML-based Simple Object Access Protocol (SOAP) can be utilized—commonlyas a foundation layer of a web services protocol stack. (Otherservice-based technologies, such as Jini, Common Object Request BrokerArchitecture (CORBA), Representational State Transfer (REST) andMicrosoft's Windows Communication Foundation (WCF) are also suitable.)

Orchestration of web services can be accomplished using the Web ServiceBusiness Process Execution Language 2.0 (WS-BPEL 2.0). Choreography canemploy W3C's Web Service Choreography Description Language (WS-CDL).JBoss's jBPM product is an open source platform adapted for use withboth WM-BPEL 2.0 and WS-CDL. Active Endpoints offers an open sourcesolution for WS-BPEL 2.0 under the name ActiveBPEL; pi4SOA onSourceForge is an open-source implementation of WS-CDL. Security for webservices can be provided through use of the WS-Security (WSS)communications protocol, a popular Java library implementation of whichis Apache's WSS4J.

Certain implementations of the present technology make use of existinglibraries of image processing functions (software). These includeCMVision (from Carnegie Mellon University—particularly good at colorimage segmentation), ImageJ (a freely distributable package of Javaroutines developed by the National Institutes of Health; see, e.g.,en<dot>Wikipedia<dot>org/wiki/ImageJ), and OpenCV (a package developedby Intel; see, e.g., en<dot>Wikipedia<dot>org/wiki/OpenCV, and the bookBradski, Learning OpenCV, O'Reilly, 2008). Well regarded commercialvision library packages include Vision Pro, by Cognex, and the MatroxImaging Library.

The refresh rate at which repeated operations are undertaken depends oncircumstances, including the computing context (battery capacity, otherprocessing demands, etc.). Some image processing operations may beundertaken for every captured frame, or nearly so (e.g., checkingwhether a lens cap or other obstruction blocks the camera's view).Others may be undertaken every third frame, tenth frame, thirtiethframe, hundredth frame, etc. Or these operations may be triggered bytime, e.g., every tenth second, half second, full second, three seconds,etc. Or they may be triggered by change in the captured scene, etc.Different operations may have different refresh rates—with simpleoperations repeated frequently, and complex operations less so.

As noted earlier, image data (or data based on image data), may bereferred to the cloud for analysis. In some arrangements this is done inlieu of local device processing (or after certain local deviceprocessing has been done). Sometimes, however, such data can be passedto the cloud and processed both there and in the local devicesimultaneously. The cost of cloud processing is usually small, so theprimary cost may be one of bandwidth. If bandwidth is available, theremay be little reason not to send data to the cloud, even if it is alsoprocessed locally. In some cases the local device may return resultsfaster; in others the cloud may win the race. By using both,simultaneously, the user can always be provided the quicker of the tworesponses. (And, as noted, if local processing bogs down or becomesunpromising, it may be curtailed. Meanwhile, the cloud process maycontinue to churn—perhaps yielding results that the local device neverprovides.) Additionally, a cloud service provider such as Google mayglean other benefits from access to the cloud-based data processingopportunity, e.g., learning details of a geographical environment aboutwhich its data stores are relatively impoverished (subject, of course,to appropriate privacy safeguards).

Sometimes local image processing may be suspended, and resumed later.One such instance is if a telephone call is made, or received; thedevice may prefer to apply its resources exclusively to serving thephone call. The phone may also have a UI control by which the user canexpressly direct the phone to pause image processing. In some suchcases, relevant data is transferred to the cloud, which continues theprocessing, and returns the results to the phone.

If local image processing does not yield prompt, satisfactory results,and the subject of the imagery continues to be of interest to the user(or if the user does not indicate otherwise), the imagery may bereferred to the cloud for more exhaustive, and lengthy, analysis. Abookmark or the like may be stored on the smart phone, allowing the userto check back and learn the results of such further analysis. Or theuser can be alerted if such further analysis reaches an actionableconclusion.

It will be understood that decision-making involved in operation of thedetailed technology can be implemented in a number of different ways.One is by scoring. Parameters associated with relevant inputs fordifferent alternatives are provided, and are combined, weighted andsummed in different combinations, e.g., in accordance with a polynomialequation. The alternative with the maximum (or minimum) score is chosen,and action is taken based on that alternative. In other arrangements,rules-based engines can be employed. Such arrangements are implementedby reference to stored data expressing conditional rules, e.g., IF(condition(s)), THEN action(s), etc. Adaptive models can also beemployed, in which rules evolve, e.g., based on historical patterns ofusage. Heuristic approaches can also be employed. The artisan willrecognize that still other decision processes may be suited toparticular circumstances.

Location-based technologies can be included to advantageous effect inmany embodiments. GPS is one such technology. Others rely on radiosignaling of the sort that that commonly occurs between devices (e.g.,WiFi, cellular, broadcast television). Patent publications WO08/073347,US20090213828, US20090233621, US20090313370, and US20100045531 describehow, given several devices, the signals themselves—and the imperfectdigital clock signals that control them—form a reference system fromwhich both highly accurate time and position information can beabstracted.

Template matching arrangements can be used in many different aspects ofthe technology. In addition to applications such as discerning likelyuser intent, and determining appropriate systems responses, based oncertain context data, template matching can also be used in applicationssuch as recognizing features in content (e.g., faces in imagery).

Template data can be stored in cloud, and refined through use. It can beshared among several users. A system according to the present technologycan consult multiple templates, e.g., of several of the user's friends,in deciding how to understand, or act in view of, incoming data.

In the particular application of content feature detection, a templatemay take the form of mask data with which unknown imagery is convolvedat different locations to find the highest output (sometimes termedLinear Spatial Filtering). Of course, the template needn't operate inthe pixel domain; the sought-for feature pattern can be defined in thefrequency domain, or other domain that is insensitive to certaintransformations (e.g., scale, rotation, color). Or multiple templatescan be tried—each differently transformed, etc.

Just as template matching can be used in many different aspects of thepresent technology, so too can the related science of probabilisticmodeling, such as in assessing the actual user context based on sensordata (e.g., eye/mouth patterns are more likely found on a face than atree), in determining appropriate responses in view of context, etc.

In certain embodiments, captured imagery is examined for colorfulness(e.g., color saturation). This may be done by converting red/green/bluesignals from the camera into another representation in which color isrepresented separately from luminance (e.g., CIELAB). In this latterrepresentation, the imagery can be examined to determine whether all—ora significant spatial area (e.g., more than 40%, or 90%)—of the imageframe is notably low in color (e.g., saturation less than 30%, or 5%).If this condition is met, then the system can infer that it is likelylooking at printed material, such as barcode or text, and can activaterecognition agents tailored to such materials (e.g., barcode decoders,optical character recognition processes, etc). Similarly, this low-colorcircumstance can signal that the device need not apply certain otherrecognition techniques, e.g., facial recognition and watermark decoding.

Contrast is another image metric that can be applied similarly (e.g.,printed text and barcodes are high contrast). In this case, a contrastmeasurement (e.g., RMS contrast, Weber contrast, etc.) in excess of athreshold value can trigger activation of barcode- and text-relatedagents, and can bias other recognition agents (e.g., facial recognitionand watermark decoding) towards not activating.

Conversely, if captured imagery is high in color, or low in contrast,this can bias barcode and OCR agents not to activate, and can insteadbias facial recognition and watermark decoding agents towardsactivating.

Thus, gross image metrics can be useful discriminants, or filters, inhelping decide what different types of processing should be applied tocaptured imagery.

Artisans implementing systems according to the present specification arepresumed to be familiar with the various technologies involved.

An emerging field of radio technology is termed “cognitive radio.”Viewed through that lens, the present technology might be entitled“cognitive imaging.” Adapting a description from cognitive radio, thefield of cognitive imaging may be regarded as “The point in whichwireless imaging devices and related networks are sufficientlycomputationally intelligent in the extraction of imaging constructs insupport of semantic extraction and computer-to-computer communicationsto detect user imaging needs as a function of user context, and toprovide imaging services wirelessly in a fashion most appropriate tothose needs.”

While this disclosure has detailed particular ordering of acts andparticular combinations of elements in the illustrative embodiments, itwill be recognized that other methods may re-order acts (possiblyomitting some and adding others), and other combinations may omit someelements and add others, etc.

Although disclosed as complete systems, sub-combinations of the detailedarrangements are also separately contemplated.

Reference was made to the internet in certain embodiments. In otherembodiments, other networks—including private networks of computers—canbe employed also, or instead.

While detailed primarily in the context of systems that perform imagecapture and processing, corresponding arrangements are equallyapplicable to systems that capture and process audio, or other stimuli(e.g., touch, smell, motion, orientation, temperature, humidity,barometric pressure, trace chemicals, etc.). Some embodiments canrespond to plural different types of stimuli.

Consider FIG. 18, which shows aspects of an audio scene analyzer (fromKubota, et al, Design and Implementation of 3D Auditory SceneVisualizer—Towards Auditory Awareness With Face Tracking, 10^(th) IEEEMultimedia Symp., pp. 468-476, 2008). The Kubota system captures 3Dsounds with a microphone array, localizes and separates sounds, andrecognizes the separated sounds by speech recognition techniques. Javavisualization software presents a number of displays. The first box inFIG. 18 shows speech events from people, and background music, along atimeline. The second box shows placement of the sound sources relativeto the microphone array at a selected time point. The third box allowsdirectional filtering so as to remove undesired sound sources. Thefourth box allows selection of a particular speaker, and a transcriptionof that speaker's words. User interaction with these displays isachieved by face tracking, e.g., moving closer to the screen and towardsa desired speaker allows the user to choose and filter that speaker'sspeech.

In the context of the present technology, a system can provide a commonvisualization of a 3D auditory scene using arrangements analogous to theSpatial Model component for camera-based systems. Baubles can be placedon identified audio sources as a function of position, time and/orclass. The user may be engaged in segmenting the audio sources throughinteraction with the system—enabling the user to isolate those soundsthey want more information on. Information can be provided, for example,about background music, identifying speakers, locating the source ofaudio, classifying by genre, etc. Existing cloud-based services (e.g.,popular music recognition services, such as from Shazam, Gracenote andMidomi) can be adapted to provide some of the audioidentification/classification in such arrangements.

In a university lecture context, a student's mobile device may capturethe voice of the professor, and some incidental side conversations ofnearby students. Distracted by colorful details of the sideconversation, the student may have momentarily missed part of thelecture. Sweeping a finger across the phone screen, the student goesback about 15 seconds in time (e.g., 5 seconds per frame), to a screenshowing various face baubles. Recognizing the face bauble correspondingto the professor, the student taps it, and transcribed text from onlythe professor's voice is then presented (and/or audiblyrendered)—allowing the student to catch what had been missed. (To speedreview, the rendering may skip over, or shorten, pauses in theprofessor's speech. Shortening may be by a percentage, e.g., 50%, or itcan trim every pause longer than 0.5 seconds down to 0.5 seconds.) Or,the student may simply swipe the professor's bauble to the top of thescreen—storing a bookmark to that location in stored audio data of thespeaker, the contents of which the student can then review later.

To perform sound source localization, two or more microphones aredesirably used. The Nexus phone handset by Google, the Droid phonehandset by Motorola, and the Apple iPhone 4 are equipped with twomicrophones, albeit not for this purpose. (The multiple microphones areemployed in active noise-cancelling arrangements.) Thus, these handsetscan be adapted to perform sound source location (as well as sound sourcerecognition) through use of appropriate software in conjunction with thesecond audio sensor. (The second audio sensor in each is amicromechanical MEMs microphone. Such devices are becoming increasinglycommon in phone handsets. Illustrative multi-microphone sound sourcelocation systems are detailed in publications US20080082326 andUS20050117754).

Additional information on sound source recognition is found, e.g., inMartin, Sound Source Recognition: A Theory and Computational Model, PhDThesis, MIT, June, 1999. Additional information on sound source locationis found, e.g., in publications US20040240680 and US20080181430. Suchtechnology can be combined with facial recognition and/or speechrecognition technologies in certain embodiments.

Additional information about distinguishing, e.g., speech from music andother audio is detailed in U.S. Pat. No. 6,424,938 and in published PCTpatent application WO08143569 (based on feature extraction).

While the detailed embodiments are described as being relatively generalpurpose, others may be specialized to serve particular purposes orknowledge domains. For example, one such system may be tailored tobirdwatchers, with a suite of image and sound recognition agentsparticularly crafted to identify birds and their calls, and to updatecrowdsourced databases of bird sightings, etc. Another system mayprovide a collection of diverse but specialized functionality. Forexample, a device may include a Digimarc-provided recognition agent toread printed digital watermarks, a LinkMe Mobile recognition agent toread barcodes, an AlpVision recognition agent to decode authenticationmarkings from packaging, a Shazam- or Gracenote music recognition agentto identify songs, a Nielsen recognition agent to recognize televisionbroadcasts, an Arbitron recognition agent to identify radio broadcasts,etc., etc. (In connection with recognized media content, such a systemcan also provide other functionality, such as detailed in applications12/271,772 (published as US20100119208) and Ser. No. 12/490,980.)

The detailed technology can be used in conjunction with video dataobtained from the web, such as User Generated Content (UGC) obtainedfrom YouTube<dot>com. By arrangements like that detailed herein, thecontent of video may be discerned, so that appropriate ad/contentpairings can be determined, and other enhancements to the users'experience can be offered. In particular, applicants contemplate thatthe technology disclosed herein can be used to enhance and extend theUGC-related systems detailed in published patent applications20080208849 and 20080228733 (Digimarc), 20080165960 (TagStory),20080162228 (Trivid), 20080178302 and 20080059211 (Attributor),20080109369 (Google), 20080249961 (Nielsen), and 20080209502(MovieLabs).

It will be recognized that the detailed processing of content signals(e.g., image signals, audio signals, etc.) includes the transformationof these signals in various physical forms. Images and video (forms ofelectromagnetic waves traveling through physical space and depictingphysical objects) may be captured from physical objects using cameras orother capture equipment, or generated by a computing device. Similarly,audio pressure waves traveling through a physical medium may be capturedusing an audio transducer (e.g., microphone) and converted to anelectronic signal (digital or analog form). While these signals aretypically processed in electronic and digital form to implement thecomponents and processes described above, they may also be captured,processed, transferred and stored in other physical forms, includingelectronic, optical, magnetic and electromagnetic wave forms. Thecontent signals are transformed in various ways and for various purposesduring processing, producing various data structure representations ofthe signals and related information. In turn, the data structure signalsin memory are transformed for manipulation during searching, sorting,reading, writing and retrieval. The signals are also transformed forcapture, transfer, storage, and output via display or audio transducer(e.g., speakers).

The reader will note that different terms are sometimes used whenreferring to similar or identical components, processes, etc. This isdue, in part, to development of this technology over time, and withinvolvement of several people.

Elements and teachings within the different embodiments disclosed in thepresent specification are also meant to be exchanged and combined.

References to FFTs should be understood to also include inverse FFTs,and related transforms (e.g., DFT, DCT, their respective inverses,etc.).

Reference has been made to SIFT which, as detailed in certain of theincorporated-by-reference documents, performs a pattern-matchingoperation based on scale-invariant features. SIFT data serves,essentially, as a fingerprint by which an object can be recognized.

In similar fashion, data posted to the blackboard (or other shared datastructure) can also serve as a fingerprint—comprisingvisually-significant information characterizing an image or scene, bywhich it may be recognized. Likewise with a video sequence, which canyield a blackboard comprised of a collection of data, both temporal andexperiential, about stimuli the user device is sensing. Or theblackboard data in such instances can be further distilled, by applyinga fingerprinting algorithm to it, generating a generally unique set ofidentification data by which the recently captured stimuli may beidentified and matched to other patterns of stimuli. (Picasso long agoforesaw that a temporal, spatially jumbled set of image elementsprovides knowledge relevant to a scene, by which its essence may beunderstood.)

As noted, artificial intelligence techniques can play an important rolein embodiments of the present technology. A recent entrant into thefield is the Alpha product by Wolfram Research. Alpha computes answersand visualizations responsive to structured input, by reference to aknowledge base of curated data Information gleaned from arrangementsdetailed herein can be presented to the Wolfram Alpha product to provideresponsive information back to the user. In some embodiments, the useris involved in this submission of information, such as by structuring aquery from terms and other primitives gleaned by the system, byselecting from among a menu of different queries composed by the system,etc. In other arrangements, this is handled by the system. Additionally,or alternatively, responsive information from the Alpha system can beprovided as input to other systems, such as Google, to identify furtherresponsive information. Wolfram's patent publications 20080066052 and20080250347 further detail aspects of the Alpha technology, which is nowavailable as an iPhone app.

Another adjunct technology is Google Voice, which offers a number ofimprovements to traditional telephone systems. Such features can be usedin conjunction with the present technology.

For example, the voice to text transcription services offered by GoogleVoice can be employed to capture ambient audio from the speaker'senvironment using the microphone in the user's smart phone, and generatecorresponding digital data (e.g., ASCII information). The system cansubmit such data to services such as Google or Wolfram Alpha to obtainrelated information, which the system can then provide back to theuser—either by a screen display, by voice (e.g., by known text-to-speechsystems), or otherwise. Similarly, the speech recognition afforded byGoogle Voice can be used to provide a conversational user interface tosmart phone devices, by which features of the technology detailed hereincan be selectively invoked and controlled by spoken words.

In another aspect, when a user captures content (audio or visual) with asmart phone device, and a system employing the presently disclosedtechnology returns a response, the response information can be convertedfrom text to speech, and delivered to the user, e.g., to the user'svoicemail account in Google Voice. The user can access this datarepository from any phone, or from any computer. The stored voice mailcan be reviewed in its audible form, or the user can elect instead toreview a textual counterpart, e.g., presented on a smart phone orcomputer screen.

(Aspects of the Google Voice technology are detailed in patentapplication 20080259918.)

Audio information can sometimes aid in understanding visual information.Different environments are characterized by different sound phenomena,which can serve as clues about the environment. Tire noise and enginesounds may characterize an in-vehicle or roadside environment. The droneof an HVAC blower, or keyboard sounds, may characterize an officeenvironment. Bird and wind-in-tree noises may signal the outdoors.Band-limited, compander-processed, rarely-silent audio may suggest thata television is playing nearby—perhaps in a home. The recurrent sound ofbreaking water waves suggests a location at a beach.

Such audio location clues can serve various roles in connection withvisual image processing. For example, they can help identify objects inthe visual environment. If captured in the presence of office-likesounds, an image depicting a seemingly-cylindrical object is more likelyto be a coffee mug or water bottle than a tree trunk. A roundish objectin a beach-audio environment may be a tire, but more likely is aseashell.

Utilization of such information can take myriad forms. One particularimplementation seeks to establish associations between particularobjects that may be recognized, and different (audio) locations. Alimited set of audio locations may be identified, e.g., indoors oroutdoors, or beach/car/office/home/indeterminate. Different objects canthen be given scores indicating the relative likelihood of being foundin such environment (e.g., in a range of 0-10). Such disambiguation datacan be kept in a data structure, such as a publicly-accessible databaseon the internet (cloud). Here's a simple example, for theindoors/outdoors case:

Indoors Score Outdoors Score Seashell 6 8 Telephone 10 2 Tire 4 5 Tree 310 Water bottle 10 6 . . . . . . . . . (Note that the indoors andoutdoors scores are not necessarily inversely related; some objects maybe of a sort likely found in both environments.)

If a cylindrical-seeming object is discerned in an image frame, and—fromavailable image analysis—is ambiguous as to whether it is a tree trunkor water bottle, reference can then be made to the disambiguation data,and information about the auditory environment. If the auditoryenvironment has attributes of “outdoors” (and/or is lacking attributesof being “indoors”), then the outdoor disambiguation scores forcandidate objects “tree” and “water bottle” are checked. The outdoorscore for “tree” is 10; the outdoor score for “water bottle” is 8, sothe toss-up is decided in favor of “tree.”

Recognition of auditory environments can be performed using techniquesand analysis that are audio counterparts to the image analysisarrangements described elsewhere in this specification. Or othertechniques can be used. Often, however, recognition of auditoryenvironments is uncertain. This uncertainty can be factored into use ofthe disambiguation scores.

In the example just-given, the audio captured from the environment mayhave some features associated with indoor environments, and somefeatures associated with outdoor environments. Audio analysis may thusconclude with a fuzzy outcome, e.g., 60% chance it is outdoors, 40%chance it is indoors. (These percentages may add to 100%, but need not;in some cases they may sum to more or less.) These assessments can beused to influence assessment of the object disambiguation scores.

Although there are many such approaches, one is to weigh the objectdisambiguation scores for the candidate objects with the audioenvironment uncertainty by simple multiplication, such as shown by thefollowing table:

Indoors score * Outdoors score * Indoors probability Outdoorsprobability (40%) (60%) Tree 3 * 0.4 = 1.2 10 * 0.6 = 6 Water bottle10 * 0.4 = 4 6 * 0.6 = 3.6

In this case, the disambiguation data is useful in identifying theobject, even through the auditory environment is not known with a highdegree of certainty.

In the example just-given, the visual analysis—alone—suggested twocandidate identifications with equal probabilities: it could be a tree,it could be a water bottle. Often the visual analysis will determineseveral different possible identifications for an object—with one moreprobable than the others. The most probable identification may be usedas the final identification. However, the concepts noted herein can helprefine such identification—sometimes leading to a different finalresult.

Consider a visual analysis that concludes that the depicted object is40% likely to be a water bottle and 30% likely to be a tree (e.g., basedon lack of visual texture on the cylindrical shape). This assessment canbe cascaded with the calculations noted above—by a furthermultiplication with the object probability determined by visual analysisalone:

Indoors score * Indoors Outdoors score * Outdoors probability (40%) *probability (60%) * Object probability Object probability Tree (30%) 3 *0.4 * 0.3 = 0.36 10 * 0.6 * 0.3 = 1.8 Water bottle 10 * 0.4 * 0.4 = 1.66 * 0.6 * .4 = 1.44 (40%)

In this case, the object may be identified as a tree (1.8 is the highestscore)—even though image analysis alone concluded the shape was mostlikely a water bottle.

These examples are somewhat simplistic in order to illustrate theprinciples at work; in actual practice more complex mathematical andlogical operations will doubtless be used.

While these examples have simply shown two alternative objectidentifications, in actual implementation, identification of one type ofobject from a field of many possible alternatives can similarly beperformed.

Nothing has yet been said about compiling the disambiguation data, e g,associating different objects with different environments. While thiscan be a large undertaking, there are a number of alternativeapproaches.

Consider video content sites such as YouTube, and image content sitessuch as Flickr. A server can download still and video image files fromsuch sources, and apply known image analysis techniques to identifycertain objects shown within each—even though many objects may gounrecognized. Each file can be further analyzed to visually guess a typeof environment in which the objects are found (e.g., indoors/outdoors;beach/office/etc.) Even if only a small percentage of videos/images giveuseful information (e.g., identifying a bed and a desk in one indoorsvideo; identifying a flower in an outdoor photo, etc.), and even if someof the analysis is incorrect, in the aggregate, a statistically usefulselection of information can be generated in such manner.

Note that in the arrangement just-discussed, the environment may beclassified by reference to visual information alone. Walls indicate anindoor environment; trees indicate an outdoor environment, etc. Soundmay form part of the data mining, but this is not necessary. In otherembodiments, a similar arrangement can alternatively—oradditionally—employ sound analysis for content and environmentcharacterization.

YouTube, Flickr and other content sites also include descriptivemetadata (e.g., keywords, geolocation information, etc.), which can alsobe mined for information about the depicted imagery, or to otherwise aidin recognizing the depicted objects (e.g., deciding between possibleobject identifications). Earlier referenced documents, includingPCT/US09/54358 (published as WO2010022185), detail a variety of sucharrangements.

Audio information can also be used to help decide which types of furtherimage processing operations should be undertaken (i.e., beyond a routineset of operations). If the audio suggests an office environment, thismay suggest that text OCR-related operations might be relevant. Thedevice may thus undertake such operations whereas, if in another audioenvironment (e.g., outdoors), the device may not have undertaken suchoperations.

Additional associations between objects and their typical environmentsmay be gleaned by natural language processing of encyclopedias (e.g.,Wikipedia) and other texts. As noted elsewhere, U.S. Pat. No. 7,383,169describes how dictionaries and other large works of language can beprocessed by NLP techniques to compile lexical knowledge bases thatserve as formidable sources of such “common sense” information about theworld. By such techniques a system can associate, e.g., the subject“mushroom” with the environment “forest” (and/or “supermarket”);“starfish” with “ocean,” etc. Another resource is Cyc—an artificialintelligence project that has assembled a large ontology and knowledgebase of common sense knowledge. (OpenCyc is available under an opensource license.)

Compiling the environmental disambiguation data can also make use ofhuman involvement. Videos and imagery can be presented to human viewersfor assessment, such as through use of Amazon's Mechanical Turk Service.Many people, especially in developing countries, are willing to providesubjective analysis of imagery for pay, e.g., identifying depictedobjects, and the environments in which they are found.

The same techniques can be employed to associate different sounds withdifferent environments (ribbetting frogs with ponds; aircraft engineswith airports; etc.). Speech recognition—such as performed by GoogleVoice, Dragon Naturally Speaking, ViaVoice, etc. (including MechanicalTurk), can also be employed to recognize the environment, or anenvironmental attribute. (“Please return your seat backs and trays totheir upright and locked positions . . . ” indicates an airplaneenvironment.)

While the particular arrangement just-detailed used audio information todisambiguate alternative object identifications, audio information canbe used in many other different ways in connection with image analysis.For example, rather than a data structure identifying the scoredlikelihoods of encountering different objects in different environments,the audio may be used simply to select one of several differentglossaries (or assemble a glossary) of SIFT features (SIFT is discussedelsewhere). If the audio comprises beach noises, the object glossary cancomprise only SIFT features for objects found near beaches (seashells,not staplers). The universe of candidate objects looked-for by the imageanalysis system may thus be constrained in accordance with the audiostimulus.

Audio information can thus be employed in a great many ways in aid ofimage analysis—depending on the requirements of particular applications;the foregoing are just a few.

Just as audio stimulus can help inform analysis/understanding ofimagery, visual stimulus can help inform analysis/understanding ofaudio. If the camera senses bright sunlight, this suggests an outdoorsenvironment, and analysis of captured audio may thus proceed withreference to a library of reference data corresponding to the outdoors.If the camera senses regularly flickering illumination with a colorspectrum that is characteristic of fluorescent lighting, an indoorenvironment may be assumed. If an image frame is captured with blueacross the top, and highly textured features below, an outdoor contextmay be assumed. Analysis of audio captured in these circumstances canmake use of such information. E.g., a low level background noise isn'tan HVAC blower—it is likely wind; the loud clicking isn't keyboardnoises; it is more likely a chiding squirrel.

Just as YouTube and Flickr provide sources for image information, thereare many freely available sources for audio information on the internet.One, again, is YouTube. There are also online libraries of sound effects(e.g., soundeffect<dot>com, sounddog<dot>com, soundsnap<dot>com, etc)that offer free, low fidelity counterparts of their retail offerings.These are generally presented in well-organized taxonomies, e.g.,Nature:Ocean:SurfGullsAndShipHorn;Weather:Rain:HardRainOnConcreteInTheCity; Transportation: Train:CrowdedTrainInterior, etc. The descriptive text data can be mined todetermine the associated environment.

Although the foregoing discussion focused on the interplay between audioand visual stimulus, devices and methods according to the presenttechnology can employ such principles with all manner of stimuli andsensed data temperature, location, magnetic field, smell, trace chemicalsensing, etc.

Regarding magnetic field, it will be recognized that smart phones areincreasingly being provided with magnetometers, e.g., for electroniccompass purposes. Such devices are quite sensitive—since they need to beresponsive to the subtle magnetic field of the Earth (e.g., 30-60microTeslas, 0.3-0.6 Gauss). Emitters of modulated magnetic fields canbe used to signal to a phone's magnetometer, e.g., to communicateinformation to the phone.

The Apple iPhone 3Gs has a 3-axis Hall-effect magnetometer (understoodto be manufactured by Asahi Kasei), which uses solid state circuitry toproduce a voltage proportional to the applied magnetic field, andpolarity. The current device is not optimized for high speed datacommunication, although future implementations may prioritize suchfeature. Nonetheless, useful data rates may readily be achieved. Unlikeaudio and visual input, the phone does not need to be oriented in aparticular direction in order to optimize receipt of magnetic input (dueto the 3D sensor). Nor does the phone even need to be removed from theuser's pocket or purse.

In one arrangement, a retail store may have a visual promotional displaythat includes a concealed electromagnet driven with a time-varyingsignal. This time-varying signal serves to send data to nearby phones.The data may be of any type. It can provide information to amagnetometer-driven smart phone application that presents a couponusable by recipients, e.g., for one dollar off the promoted item.

The magnetic field data may simply alert the phone to the availabilityof related information sent through a different communication medium. Ina rudimentary application, the magnetic field data can simply signal themobile device to turn on a specified input component, e.g., BlueTooth,NFC, WiFi, infrared, camera, microphone, etc. The magnetic field datacan also provide key, channel, or other information useful with thatmedium.

In another arrangement, different products (or shelf-mounted devicesassociated with different products) may emit different magnetic datasignals. The user selects from among the competing transmissions bymoving the smart phone close to a particular product. Since the magneticfield falls off in exponential proportion to the distance from theemitter, it is possible for the phone to distinguish the strongest(closest) signal from the others.

In still another arrangement, a shelf-mounted emitter is not normallyactive, but becomes active in response to sensing a user, or a userintention. It may include a button or a motion sensor, which activatesthe magnetic emitter for five-fifteen seconds. Or it may include aphotocell responsive to a change in illumination (brighter or darker).The user may present the phone's illuminated screen to the photocell (orshadow it by hand), causing the magnetic emitter to start a five secondbroadcast. Etc.

Once activated, the magnetic field can be utilized to inform the userabout how to utilize other sensors that need to be positioned or aimedin order to be used, e.g., such as cameras, NFC, or microphones. Theinherent directionality and sensitivity to distance make the magneticfield data useful in establishing the target's direction, and distance(e.g., for pointing and focusing a camera). For example, the emitter cancreate a coordinate system that has a package at a known location (e.g.,the origin), providing ground-truth data for the mobile device.Combining this with the (commonly present) mobile deviceaccelerometers/gyroscopes, enables accurate pose estimation.

A variety of applications for reading barcodes or other machine readabledata from products, and triggering responses based thereon, have beenmade available for smart phones (and are known from the patentliterature, e.g., US20010011233, US20010044824, US20020080396,US20020102966, U.S. Pat. No. 6,311,214, U.S. Pat. No. 6,448,979, U.S.Pat. No. 6,491,217, and U.S. Pat. No. 6,636,249). The same arrangementscan be effected using magnetically sensed information, using a smartphone's magnetometer.

In other embodiments, the magnetic field may be used in connection withproviding micro-directions. For example, within a store, the magneticsignal from an emitter can convey micro-directions to a mobile deviceuser, e.g., “Go to aisle 7, look up to your left for product X, now onsale for $Y, and with $2 additional discount to the first 3 people tocapture a picture of the item” (or of a related promotional display).

A related application provides directions to particular products withina store. The user can key-in, or speak, the names of desired products,which are transmitted to a store computer using any of various signalingtechnologies. The computer identifies the locations of the desiredproducts within the store, and formulates direction data to guide theuser. The directions may be conveyed to the mobile device magnetically,or otherwise. A magnetic emitter, or a network of several emitters,helps in guiding the user to the desired products.

For example, an emitter at the desired product can serve as a homingbeacon. Each emitter may transmit data in frames, or packets, eachincluding a product identifier. The original directions provided to theuser (e.g., go left to find aisle 7, then halfway down on your right)can also provide the store's product identifiers for the productsdesired by the user. The user's mobile device can use these identifiersto “tune” into the magnetic emissions from the desired products. Acompass, or other such UI, can help the user find the precise locationof the product within the general area indicated by the directions. Asthe user finds each desired product, the mobile device may no longertune to emissions corresponding to that product.

The aisles and other locations in the store may have their ownrespective magnetic emitters. The directions provided to the user can beof the “turn by turn” variety popularized by auto navigation systems.(Such navigation technologies can be employed in other embodiments aswell.) The mobile device can track the user's progress through thedirections by sensing the emitters from the various waypoints along theroute, and prompt the user about next step(s). In turn, the emitters maysense proximity of the mobile device, such as by Bluetooth or othersignaling, and adapt the data they signal in accord with the user andthe user's position.

To serve multiple users, the transmissions from certain networks ofemitters (e.g., navigational emitters, rather than product-identifyingemitters) can be time-division multiplexed, sending data in packets orframes, each of which includes an identifier indicating an intendedrecipient. This identifier can be provided to the user in response tothe request for directions, and allows the user's device to distinguishtransmissions intended for that device from others.

Data from such emitters can also be frequency-division multiplexed,e.g., emitting a high frequency data signal for one application, and alow frequency data signal for another.

The magnetic signal can be modulated using any known arrangementincluding, but not limited to, frequency-, amplitude-, minimum- orphase-shift keying, quadrature amplitude modulation, continuous phasemodulation, pulse position modulation, trellis modulation, chirp- ordirect sequence-spread spectrum, etc. Different forward error correctioncoding schemes (e.g., turbo, Reed-Solomon, BCH) can be employed toassure accurate, robust, data transmission. To aid in distinguishingsignals from different emitters, the modulation domain can be dividedbetween the different emitters, or classes or emitters, in a manneranalogous to the sharing of spectrum by different radio stations.

The mobile device can be provided with a user interface especiallyadapted for using the device's magnetometer for the applicationsdetailed herein. It may be akin to familiar WiFi userinterfaces—presenting the user with information about availablechannels, and allowing the user to specify channels to utilize, and/orchannels to avoid. In the applications detailed above, the UI may allowthe user to specify what emitters to tune to, or what data to listenfor—ignoring others.

Reference was made to touchscreen interfaces—a form of gestureinterface. Another form of gesture interface that can be used inembodiments of the present technology operates by sensing movement of asmart phone—by tracking movement of features within captured imagery.Further information on such gestural interfaces is detailed inDigimarc's patent 6,947,571. Gestural techniques can be employedwhenever user input is to be provided to the system.

Looking further ahead, user interfaces responsive to facial expressions(e.g., blinking, etc) and/or biometric signals detected from the user(e.g., brain waves, or EEGs) can also be employed. Such arrangements areincreasingly well known; some are detailed in patent documents20010056225, 20020077534, 20070185697, 20080218472 and 20090214060. Thephone's camera system (and auxiliary cloud resources) can be employed torecognize such inputs, and control operation accordingly.

The present assignee has an extensive history in content identificationtechnologies, including digital watermarking and fingerprint-basedtechniques. These technologies have important roles in certain visualqueries.

Watermarking, for example, is the only container-independent technologyavailable to identify discrete media/physical objects withindistribution networks. It is widely deployed: essentially all of thetelevision and radio in the United States is digitally watermarked, asare uncountable songs, motion pictures, and printed documents.

Watermark data can serve as a type of Braille for computers—guiding themwith information about a marked object (physical or electronic).Application of pattern recognition techniques to an image may, after along wait, yield an output hypothesis that the image probably depicts ashoe. In contrast, if the shoe bears digital watermark data, then in amuch shorter time a much more reliable—and accurate—set of informationcan be obtained, e.g., the image depicts a Nike basketball shoe, size11M, model “Zoom Kobe V,” manufactured in Indonesia in May 2009.

By providing an indication of object identity as an intrinsic part ofthe object itself, digital watermarks greatly facilitate mobiledevice-object interaction based on an object's identity.

Technology for encoding/decoding watermarks is detailed, e.g., inDigimarc's U.S. Pat. Nos. 6,614,914 and 6,122,403; in Nielsen's U.S.Pat. Nos. 6,968,564 and 7,006,555; and in Arbitron's U.S. Pat. Nos.5,450,490, 5,764,763, 6,862,355, and 6,845,360.

Digimarc has various other patent filings relevant to the presentsubject matter. See, e.g., patent publications 20070156726, 20080049971,and 20070266252.

Examples of audio fingerprinting are detailed in patent publications20070250716, 20070174059 and 20080300011 (Digimarc), 20080276265,20070274537 and 20050232411 (Nielsen), 20070124756 (Google), U.S. Pat.No. 7,516,074 (Auditude), and U.S. Pat. Nos. 6,990,453 and 7,359,889(both Shazam). Examples of image/video fingerprinting are detailed inU.S. Pat. No. 7,020,304 (Digimarc), U.S. Pat. No. 7,486,827(Seiko-Epson), 20070253594 (Vobile), 20080317278 (Thomson), and20020044659 (NEC).

Nokia acquired a Bay Area startup founded by Philipp Schloter that dealtin visual search technology (Pixto), and has continued work in that areain its “Point & Find” program. This work is detailed, e.g., in publishedpatent applications 20070106721, 20080071749, 20080071750, 20080071770,20080071988, 20080267504, 20080267521, 20080268876, 20080270378,20090083237, 20090083275, and 20090094289. Features and teachingsdetailed in these documents are suitable for combination with thetechnologies and arrangements detailed in the present application, andvice versa.

In the interest of conciseness, the myriad variations and combinationsof the described technology are not cataloged in this document.Applicants recognize and intend that the concepts of this specificationcan be combined, substituted and interchanged—both among and betweenthemselves, as well as with those known from the cited prior art.Moreover, it will be recognized that the detailed technology can beincluded with other technologies—current and upcoming—to advantageouseffect.

To provide a comprehensive disclosure without unduly lengthening thisspecification, applicants incorporate-by-reference the documents andpatent disclosures referenced above. (Such documents are incorporated intheir entireties, even if cited above in connection with specific oftheir teachings.) These references disclose technologies and teachingsthat can be incorporated into the arrangements detailed herein, and intowhich the technologies and teachings detailed herein can beincorporated.

1. A method comprising the acts: receiving sensor data, the sensor dataincluding audio data sensed using a microphone of a user's portabledevice, and image data sensed using a camera of said device; determiningdynamically, based on context, a recognition-related operation toperform on the received sensor data, from among plural differentrecognition-related operations; and determining dynamically, based oncontext, whether to make available more or less resources to saiddetermined recognition-related operation; wherein plural differentrecognition-related operations are performed on received sensor data,over time, due to different contexts; and wherein resources madeavailable to said determined recognition-related operation are changed,over time, due to changing context. 2-3. (canceled)
 4. The method ofclaim 1 wherein said changing context comprises a new operation that hasbecome ready to run, requiring reallocation of resources. 5-19.(canceled)
 20. The method of claim 1 that includes determining saidrecognition-related operation based, at least in part, on a relevancescore.
 21. The method of claim 20 that includes determining saidrecognition-related operation also based on a resource requirement. 22.The method of claim 1 that includes determining said recognition-relatedoperation based, at least in part, on a resource requirement.
 23. Themethod of claim 1 that includes determining to make more resourcesavailable to said determined recognition-related operation based, atleast in part, on implied user encouragement.
 24. The method of claim 1that includes determining to make more resources available to saiddetermined recognition-related operation based, at least in part, on adetection state assessment of said recognition-related operation. 25.The method of claim 1 that includes determining to make less resourcesavailable to said determined recognition-related operation based, atleast in part, on an output from a different recognition-relatedoperation.
 26. The method of claim 1 wherein plural different imagerecognition-related operations are performed on received sensor data,over time, due to different contexts.
 27. The method of claim 1 in whichat least one of said determining acts is based on computing context. 28.The method of claim 1 in which at least one of said determining acts isbased on user context.
 29. The method of claim 1 in which at least oneof said determining acts is based on physical context.
 30. The method ofclaim 1 in which at least one of said determining acts is based ontemporal context.
 31. The method of claim 1 in which at least one ofsaid determining acts is based on history of context.
 32. The method ofclaim 1 that includes determining said recognition-related operationusing an N-ary tree-based planning exercise.
 33. A portable wirelessdevice comprising: a battery; a camera and a microphone, said camera andmicrophone providing sensor information; a wireless interface; aprocessor; and a memory, said memory containing software instructionsthat configure the device to perform actions including: determiningdynamically, based on context, a recognition-related operation toperform on sensor information, from among plural differentrecognition-related operations; and determining dynamically, based oncontext, whether to make available more or less resources to saiddetermined recognition-related operation; wherein said instructions areoperative to cause the device to perform plural differentrecognition-related operations on sensor information, over time, due todifferent contexts; and wherein said instructions are operative to causeresources made available to said determined recognition-relatedoperation to change, over time, due to changing context.
 34. The deviceof claim 33 in which said actions include determining to make moreresources available to said determined recognition-related operationbased, at least in part, on implied user encouragement.
 35. The deviceof claim 33 in which said actions include determining to make moreresources available to said determined recognition-related operationbased, at least in part, on a detection state assessment of saidrecognition-related operation.
 36. A computer readable storage mediumfor use with a portable device that includes a camera and a microphonefor providing sensor information, said medium containing softwareinstructions that, if executed by said device, cause the device toperform actions including: determining dynamically, based on context, arecognition-related operation to perform on the sensor information, fromamong plural different recognition-related operations; and determiningdynamically, based on context, whether to make available more or lessresources to said determined recognition-related operation; wherein saidinstructions are operative to cause the device to perform pluraldifferent recognition-related operations on sensor information, overtime, due to different contexts; and wherein said instructions areoperative to cause resources made available to said determinedrecognition-related operation to change, over time, due to changingcontext.
 37. The storage medium of claim 37 in which said actionsinclude determining to make more resources available to said determinedrecognition-related operation based, at least in part, on a detectionstate assessment of said recognition-related operation.